Good Luck EDge

Good luck EDge

This is still around?

EDge 4 Prez

Any old school players still around?

Thanks for the ideas. Looking to build a script that can be run via cron on a regular basis grabbing new data. I tried using fiddler to inspect the requests while manually navigating the site without making much headway. Although tampermonkey won't work for this specific task, it looks pretty useful for other things, thanks!

I'm trying to build something that will crawl and scrape the files from https://marketplace.spp.org/...guest/binding-constraints

Haven't had much luck with the traditional methods so was hoping someone could help me out. Preferably in Python or C#, but a simple way to grab the urls for each file after you go through the folders would help too. It looks like it's using a filebrowser widget which makes the easy way of grabbing links a little tough.

Thanks!

Good luck EDge

No overland flood insurance available for residential households

Good Luck EDge!

Good Luck EDge!

Good Luck EDge!

I was planning on using the shared unlimited hosts, or the dreamhost VPS which is unlimited disk/bandwidth, to begin with just so I can gauge what kind of numbers I'll be looking at. After everything is up and running I should have enough usage data to look for other options that will provide better performance but allow for enough bandwidth/disk space.

In terms of making the system distributed, the current setup has a different console app/windows process for each logically different scrape.

I have worked with PHP, Java, C/C++, C#, VB, in the past but was planning to go with Python due to some libraries available for it which I plan to utilize for some of the modelling. However, if a specific language will be more efficient than another, I'm okay having the scrapes in one language and the models in another.

There will be maybe a dozen pages that are of the high frequency type; but if it does become a significant issue, I can just run something on one of my boxes at home to handle those. I was just hoping to have everything on a hosted server for security and accessibility.

Here's an example of something that I scrape:
www dot spp dot org/XML/LIP-Pricing dot xml

This is not one of the frequently updated sources though, this will only be revised every five minutes. And yes, I did replace . with dot lol. Just don't need this thread showing up on a google search result if a competitor searches the same url.

Thanks, think I'll just sign up for a dreamhost account.

Anyone have an idea for how I can get a scrape working that checks a site every few seconds for changes? Right now I have windows system processes take care of that, but need an alternative that will work on these linux servers. Having a cronjob that executes every few seconds doesn't seem optimal to me.

fixed

So looking over one of the tables for 2012, there are about 270M rows of data using 17GB of space in MS Sql Server. Based on that, I am starting to think I might be able to get away without an "unlimited" plan.

Xin, why is it that you guys are using Rackspace for production instead of dreamhost? Better performance? Reliability? Speed? Or do they support products that dreamhost doesnt?

Speeds pretty good with dreamhost azz kikr? How is their support team? Do you use them from apps, or plain html sites?

I'm looking at Dreamhost and hostgator right now. Either the shared hosting on either provider since they offer unlimited disk space and bandwidth, or the VPS on dreamhost since it allows unlimited bandwidth/diskspace.

Anyone have any experience with either provider, or see any issues with their plans vs what I'm planning to do?

The current servers both have 24 core amd opteron processors with 16gb of ram and raid harddrives. I know performance wise it will be a significant step down with these hosting plans, but I won't be doing nearly as much computing or queries off the webhost.

primary key will be the datetime int combo

data will only be inserted. majority of the tables will have 3 columns. Datetime, int, decimal

Also, not sure how the 5 second scrapes will work with crontab, or if there's a better approach to take for those.

The current setup that is working is with a separate SQL Server box, and an app server. The high frequency data is scraped using c# windows processes and the not as frequent data is scraped using c# console apps launched with windows task scheduler.

I would like to migrate this all to a remote host though and was thinking I could use mysql as the database, and python scripts for the scrapes executed by crontab. I am not all too familiar with any of this technology, but I figure basic data scrapes shouldn't require any real advanced programming in Python. The only thing I will need to become familiar with is db tuning the mysql database since it will have so much data.

Good Luck EDge!

So I'm looking at starting a little project. It is very data intensive as it involves scraping a lot of websites, downloading a bunch of csv's off SSL sites requiring client side certs, scraping and parsing various kml files and storing all of this data into a database.

There will be some data that is inserted just once, but majority of the data will be scraped frequently. Some change as often as every 5 seconds, majority change every 5 minutes.

I then have a few reports and models that will run based on this data.

My questions are:

1) Any recommendations for webhosts that I can use for this
2) Any recommendations for which framework/language/database that would be ideal for this
3) Anything you think I should know heading in to this

Keep in mind the volume of data isn't small. Initial load will include in excess of half a billion rows of data, with over half a mill rows added daily.

Thanks!

Good Luck EDge!

Sodawater's a chill dude. Sucks this happened to him.

Good Luck EDge

Good Luck EDge!

Good Luck EDge!

Good Luck EDge!

Good Luck EDge!

I remember you!

pank... hahaha nice. I bet majority of the people here don't know the origin of the word

Good Luck EDge

Good Luck EDge

Suits
White Collar
Californication
Sons of Anarchy
Entourage
House of Lies
Game of Thrones
Walking Dead
The Newsroom
Falling Skies

Hate Skylar so much!

.

Good Luck EDge!

Good Luck EDge

Good Luck EDge

Good Luck EDge

Good Luck EDge

Good Luck EDge

Good Luck EDge

Good Luck EDge

Good Luck EDge

Good Luck EDge

Good Luck EDge

Good Luck EDge

Is this a product you're developing? Are there any assets that you can secure the loan against? Which part of Canada are you from?