LogoCollege Data Scraper
Personal2/2020 - 6/2020

Summary
My college had a website where all the data of the students were stored. The website was horrible. It was slow, it was not user-friendly, it was not mobile-friendly as well. However, the college had no other option but to use the website, forcing the students as well as the teachers to use the website. One day, I wanted to learn something new, and decided to create a scraper for our college website. I learned a lot of new things, but the website was not easy to scrape. I had to spend a lot of time to scrape the website. I was finally able to scrape the website, and get the data of all the logged in students. It was a fun project, and stayed that way for a long time, until I found a use for it.Scenario

My college had a website where all the data of the students were stored. The data included the student's name, roll number, email, phone number, etc., along with student's marks and attendance in each subject.

The website was a 3rd party website, they had leased the website to the college. The college had no control over the website. Any changes to the website had to be requested to the 3rd party company.

The website was horrible. It was slow, it was not user-friendly, it was not mobile-friendly as well. However, the college had no other option but to use the website, forcing the students as well as the teachers to use the website.

My motivation behind the scraper

This was the start of my second year in college. New students were being inducted into the college. There was a ceremony for the new students. The ceremony was held in the college auditorium.

To motivate the new students, the college dean presented a lot of prizes to the students who had performed well in the previous year. Many of them had scored above 90% in their exams, many of them had won medals in sports, many of them had won prizes in cultural events, etc.

There were a few students, who had created amazing projects. They had created websites, mobile apps, etc. They had also won prizes for their projects. I was sad that I was not one of them. I had not created any project. I had not won any prize. I had not scored above 90% in any of my exams. I was just an average student.

Instead of feeling sad, I decided to do something about it. I decided to learn something new.

Creating a scraper

I found a new framework called Scrapy, and wanted to learn it. I scraped a few test websites, so that I could learn Scrapy quickly. It was fun. This was the first time I got to know there's something called generators in Python. I had never used them before.

After a few days, I tried scraping my college website. It was an aspx website, created using ASP.NET. And it was horrible. Everytime I tried to log in, the website would not let me. Each request would eventually redirect me to the login page. I could not understand why. I tried to log in using the browser, and then the browser would let me log in. But the scraper would not.

I decided to compare the requests. I found what was wrong. The website was setting a cookie, and the scraper was not sending the cookie back to the server. I fixed the issue, but did the login work? No. The website would still redirect me to the login page.

Why

I was frustrated. I had spent a lot of time on this. I had learned a lot of new things, but I could not scrape the website. I decided to send every single header that the browser was sending. I copied all the headers from the browser, and sent them in the request headers of the scraper. it worked.. I was able to log in. Huh? what was wrong before? I analyzed the request, experimented removing each header one by one and found the issue.

The website had a referer policy. The website would check the referer header of the request. If the referer header was not set, the website would redirect the user to the login page.

Why

Hours of frustration, and the issue was so simple. I had learned a lot of new things, but I had wasted a lot of time as well. It was finally working. I was able to scrape the website. I was able to get the data of all the logged in student.

Conclusion

I created the scraper, it was working fine. I was happy, that I had learned something new. My sadness was gone, I found a new level of confidence in me.

However, I could not find any use for the scraper. It was a fun project, but it stayed in my computer for a long time, without any use. Until........ I found a use for it. Read it here