web scraping - Making a basic web scrapper in Python with only built in libraries - Python newbie -

-

i'm quite new python, instead of reading tutorials felt completing actual project better way learn. however, means i'd needing quite assistance (:

now point.

i'm trying make web scrapper without 3rd party libraries, process isn't simplified me, , know doing. looked through several online resources, of have left me confused things.

the html looks this,

<html> <head>...</head> <body>     *lots of other <div> tags* <div class = "want" style="font-family:verdana;font-size:12px;letter-spacing:normal""> <form class ="subform">...</form> <div class = "subdiv1" >...</div> <div class = "subdiv2" >...</div>     *lots of other <div> tags* </body> </html>

i want scrapper extract <div class = "want"...>*content*</div> , save html file.

i have basic idea of how need go this.

import urllib urllib import request #import re #from html.parser import htmlparser  response = urllib.request.urlopen("http://website.com") html = response.read()  #some how extract wanted data  f = open('page.html', 'w') f.write(data) f.close()

thank much!

the standard library comes variety of structured markup processing tools, can use parsing html , searching extract div.

there's whole lot of choices there. use?

html.parser looks obvious choice, i'd start elementtree instead. it's nice , powerful api, , there's tons of documentation , sample code on web started, , lot of experts using on daily basis can problems. if turns out etree can't parse html, have use else… try first.

for example, few minor fixes snipped html it's valid, , there's text worth getting out of div:

<html> <head>...</head> <body>     *lots of other <div /> tags* <div class = "want" style="font-family:verdana;font-size:12px;letter-spacing:normal">spam spam spam <form class ="subform">...</form> <div class = "subdiv1" >...</div> <div class = "subdiv2" >...</div>     *lots of other <div /> tags* </div> </body> </html>

you can use code (i'm assuming know, or willing learn, xpath):

tree = elementtree.fromstring(page) mydiv = tree.find('.//div[@class="want"]')

now you've got reference div class "want". can direct text this:

print(mydiv.text)

but if want extract whole subtree, that's easier:

data = elementtree.tostring(mydiv)

if want wrap in valid <html> , <body> and/or remove <div> itself, you'll have part manually. documentation explains how build elements using simple tree api: create head , body put in html, stick div in body, tostring html, , that's it.


Comments

Post a Comment

Popular posts from this blog

mod rewrite - Using "?" when rewriting the URL -

-
i have been researching weeks looking how rewritting file/directory complementary "?" or "question mark" what need: http://mydomain.com/demo01.php to like: http://mydomain.com/?theme=jerus theme > simple directory need use websites products(in case themes) jerus > name of theme extra info: i plan on rewritting every url individually refrain wildcard varibles.** wouldn't easy instead of doing url rewriting put php in directory called theme , name index.php? http://mydomain.com/theme/?t=jerus but if you're re-writing urls make rule acutally make route this: http://mydomain.com/theme/jerus/ which waaaay prettier see question: using clean urls in restful api
Read more

Admob integration with pygame in android -

-
hello curious know solution integrating admob app written in pygame subset android. have tried below things did' t work me :( still nothing working me may there no way :( try 1: followed google https://developers.google.com/mobile-ads-sdk/docs/admob/fundamentals#android solution has no code change build.xml modification didt works me :( try 2: tried below solutio also. http://www.michenux.net/android-admob-tutorial-461.html did't work me :( try 3: tried hack pythonactivity.java. getting below related errors. have googleadmobadssdk-6.4.1.jar in \pgs4a-0.9.4\libs path still below kind of error. below code: protected void oncreate(bundle savedinstancestate) { super.oncreate(savedinstancestate); //vrbilgi setcontentview(r_layout_main); // setcontentview(r.layout.main); // create adview // adview = new adview(this, adsize.banner, my_ad_unit_id); adview = new adview(this, adsize.banner,"a15...
Read more

.htaccess: Transfer name to index.php if not directory public -

-
i have piece of code: options followsymlinks <ifmodule mod_rewrite.c> rewriteengine on rewritecond %{request_filename} !-f rewritecond %{request_filename} !-d rewritecond %{request_filename} !-l rewriterule ^(.*)$ index.php?url=$1 [l] </ifmodule> <ifmodule !mod_rewrite.c> errordocument 404 /index.php </ifmodule> i don't know how allow directory name "public" inside folder .htaccess, other names dir should transfered index.php. how that? add rewritecond exclude public directory rewritecond %{request_filename} !-f rewritecond %{request_filename} !-d rewritecond %{request_filename} !-l rewritecond %{request_uri} ^/public(/.*)?$ rewriterule ^(.*)$ index.php?url=$1 [l] now, redirection work /public directories only. if want url have rest of path that's below public url=subfolder/page.php /public/subfolder/page.php use rewritecond %{request_filename} !-f rewritecond %{request_filename} !-d rewrit...
Read more