Was about to scrape pages from a CC-licensed website (BY-NC-SA). Which made me think of the practice of webscraping. In my experience (as a non-coder), webscraping is something of a craft.
Who in the OER community has developed solid competencies in webscraping?
Which tools/techniques could you share for doing it properly? (e.g. do you use Beautiful Soup? Goose?)
Has there been work done on the harvesting side of the OER scene which connects to webscraping?
Is there a proper way to “fork” content this way, keeping some kind of trace of how content diverges?
For background… The site I’m about to scrape is a Google Site containing lists upon lists of tools that we can use in learning contexts. (So, a bit more “meta” than true OERs.) The site hasn’t been updated since 2014.
One thing I’m thinking of doing in the longer term is to use these lists as the basis for actual OERs by connecting the tools to competencies that learners are expected to develop.
It’s definitely an alchemy like craft, though you run the range form scraping for data collection e.g. harvesting data from multiple sites or different time slots of the same site to create something new to more of the archival approach (things like Webrecorder or Archive.is).
I am not raising my hand for having solid competencies, more of a flyby curiosity. I have used SiteSucker an OSX app for making static HTML archives of WordPress sites that I archived – but this does not sound where you are fishing.
It contains several pages with information on software used in teaching. A bunch of tables, basically.
There are other tables like these. In fact, after posting my query, I was notified of a Community of Practice’s list of Free/Libre Open Source Software by discipline.
What I’m trying to do, there, is a compilation of all of these things.
Doing it by hand would probably not take that long. I just get the “urge” to automate the process. Call it the xkcd effect. Should I Automate This? | Hackaday
Actually, I perceive it as an opportunity to develop skills.
I also realize that freeware and freemium services are mixed in with the F/LOSS. Same with the site I want to scrape.
Also for context… I’ve recently joined the board of Adte.ca, an organization promoting F/LOSS in HigherEd. Something we’ve discussed this week is using Free Software to create OERs… and using OER approaches to document Free Software.
As you might notice by now, some (many?) of my questions are half-baked thoughts and/or “messages in bottles”. Thankfully, it often generates something useful when it goes in a different direction.
This one might help with some data literacy work I’m planning. Which leads back to the connections between Open Data and Open Education.
I’ve done a decent amount of scraping with Google Sheet’s IMPORTXML function and Google Script/javascript (although I always hope for an API instead). I’ve dabbled some in Python and PHP to do it as well.
I often rely on nice little tools like the browser plugin Scraper for lightweight stuff. IMPORTHTML in Google sheets is a handy way to get tables.
I came across Spatula the other day and will likely give it a shot the next time something comes up.
Sorry for the lack of links. I can only use two as a new user.
Those solutions based on Google Sheets do sound quite interesting. Haven’t done much in Google Sheets for a while but these sound very appropriate (especially since scraping tables is the main thing I originally wanted to do).
I’ve gone back to Python for this type of thing. Since I’m not a coder, it takes me quite a bit of time and I need to be in the right “mode” to do it. Once I do get something together, it’s a lot more flexible and powerful than most off-the-shelf solutions. I currently do it in Google Colab as Jupyter Notebooks with pandas, Beautiful Soup, etc.
Though these sorts of technical questions may not bring the OE Global community together, it sounds to me like a bit of “tooling” could enhance our work.
My scraping has been driven by random faculty trying to do get data for analysis with occasional personal departure that tend to be for amusement. I also have done a number of stackoverflow questions on this over the years. That tends to be sports/stocks type of things with people wanted to do it in Google Sheets.
In general, it’s about data analysis about 95% of the time. The other pieces tend to be about improving or altering what I can do with the data. I’ll often pull out big tables and put them into spreadsheet to manipulate them more easily. Stuff like that.
Stuff I have scraped.
various things from Instagram related to public health after they shut down most of the API (vaping, ecigs and views, hashtags etc.)
CDC comments on Facebook on posts related to Ebola for racism analysis
a giant discussion board on getting green cards
twitter messages from an educational charter school (for past data when they didn’t set up TAGS ahead of time)
youtube video views, likes, dislikes for a series of videos that were not owned by the faculty member
Gangham style views on youtube
sexual assault in higher ed page
Snoop Dog’s instagram followers
a bookmarkelet to let me play Alan Lomax songs without going through an unpleasant archive interface
an export bookmarklet to get content out of Top Hat so it could be put in PressBooks
TAGS!
Haven’t heard of that in a while. It was my primary reason to use Google Sheets, at some point.
For idiosyncratic reasons, it reminds me of NodeXL…
And while I fully realize this may all be tangential to OEG, I perceive a path to something very useful about the connections between Open Data, Open Pedagogy, and Open Educational Resources (with a hint of Free/Libre Open Source Software, though Google services are notoriously very much nonfree, especially if you’ve ever heard RMS talk in public).
Basically, there’s something about dynamic resources for learning. A bit like Adaptive Learning (with Learning Analytics and such). Yet much more about opening the spigot from Open Data firehoses and having your learning material accommodate for that data influx.
Since it’s early Summer, here, I’m in the most actively divergent part of my design thinking journey. It involves disparate dimensions from curation and annotation (last week was #ianno21) all the way to forking H5P from a content bank integrated in a federated LMS.
Twitter TAGS is remarkable and Martin Hawksey a genius. He had more than was there to meet the eye.
My flip on analytics in co-teaching Networked Narratives was with having student activities in twitter, hypothes.is was asking them to evaluate and reflect on what their activity within the larger network reflected.
So we would ask students to characterize their activity and their “place” in the larger network. This turns analytics inside out (maybe), where the machine just takes the data. For even more mine blowing, for any single user’s node, click the “Replay Tweets” button. I am not sure anyone else even saw that, but it plays back your history in time.
Asking students to share their link in the network, and another link to say, their hypothes.is annotations, provided a view into their activity. Better than counting comments.
(I should say, twitter was never required, we offered students to participate in other ways if it was not a safe or viable space for them).
