Results 1 to 3 of 3

Thread: Need Direction and advise? database newbie

  1. #1
    Join Date
    May 2003
    Posts
    3

    Need Direction and advise? database newbie

    I am in the process of building a web page that is database driven. I am going to need the database to contain various data on 400 products from 200 websites. I have a couple questions but I will for now stick to one. Instead of
    going to each site and manually hand entering in each and every product and info on each can I "spider' those sites?
    And search for the data? Can this be easily done even if the 200 sites have different formats of data storage? ANd this site search would need to be done intially to put all the data on my database. and it would also need to be updated at least quarterly, if not monthly. With that said would it not be easier to request from each sites webmaster to send me the database in whatever file they have it stored in?
    And with that could I not stake that data and easily store it in my one common database? What database I am going to use I do not yet know. I was thinking of speeding up the process of colecting my content for the database by attempting to save ALL data in MS access. But I would hate to go through all that just to find out that I could not transfer the Access to a more appropriate web driven database format.

    What suggestions do you guys and gals have for me. I can answer in more detail what I want to do if need be.
    But to keep it general since data is data. Thanks.

  2. #2
    Join Date
    Sep 2002
    Location
    Fantasy
    Posts
    4,254
    1. Every website will have different kind of database engine and security, so you can't just hack in.

    2. Every website will have different formats and outputs, it is imperative that you have to analyse every website

    3. On the net there are softwares that can download files, pics etc from any url you give. But I am not sure what kind of data you are looking in to websites to download.

    The alternate way to make things easy is ask all the webmasters to send their details in a csv format or XL or whichever format is acceptable by MS-Access.

  3. #3
    Join Date
    May 2003
    Posts
    3

    Hi

    I appreciate your input , and apologize for the lack of info. I was lucky to have met a programmer that has wrote a program that just simply pulled the source code from the 200 web sites I need and all the info was within this HTML code.
    The program extracted the data I needed and placed it in the appropriate place in a data base. I was quoted @$3,000 to do this from several companies. One company said it would take a team of programmers a month. And I ran into a guy that looked at what I wanted to do and wrote the code from Visual basic in less than 18 hours.
    What it does is extract brand names, prices and quantiies, among a few other fields, and takes the data from each of their code and places each in fields exactly where I need it in my database. I honestly thought I was going to have to either pay someone $3,000 to write software to do this or else hand enter all the data and that happened to be 10's if not 100's of thousands of fields for my database extracted from 200 sites, That is a lot of fields! Now I have some software written in less than 2 days that can do all I need it to do and update my database daily.

    I really feel I have gotten very lucky to have found a super talented programmer. I am just curious why I was quoted so much from established web developer companies when it was accomplished in such a short time by a self taught programmer??

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •