Drexel Scraper MCP Server

UI for data: https://scheduler.zohair.dev

TMS-Scraper

Scrapes data from the Drexel Term Master Schedule and outputs a JSON file. Note that this scraper is not officially supported by Drexel University, and may break if the term master schedule website changes.

Installation

Make sure git is installed. Clone the repository:

git clone https://github.com/Zohair-coder/drexel-scraper.git
cd drexel-scraper

Make sure you have Python 3 installed. Then install the dependencies by running the following command:

Mac/Linux

pip3 install -r requirements.txt
playwright install

Windows

pip install -r requirements.txt
playwright install

Usage

First, set up authentication. Then, to run the scraper, simply run:

Mac/Linux

python3 src/main.py

Windows

python src/main.py

The scraper will output a JSON file called data.json in the same directory as the scraper.

Automatic Quarter Detection

The scraper now automatically detects the current Drexel quarter based on the current date. You no longer need to manually update the year and quarter values in src/config.py. The system uses the following schedule:

• Fall Quarter (15): July 1 - September 27
• Winter Quarter (25): September 28 - January 15
• Spring Quarter (35): January 16 - April 14
• Summer Quarter (45): April 15 - June 30

If you need to override the automatic detection (e.g., for testing or scraping a specific past/future quarter), you can set the DREXEL_YEAR and DREXEL_QUARTER environment variables:

export DREXEL_YEAR=2024
export DREXEL_QUARTER=35  # Spring quarter

You can still modify the college_code variable in src/config.py to scrape a specific college.

To view all the options that the scraper supports, run python3 src/main.py --help on Mac/Linux, or python src/main.py --help on Windows.

Authentication

Since the term master schedule is only accessible to logged-in Drexel students, to run the scraper, you will need to provide your Drexel credentials as well as provide multi-factor authentication (MFA).

To provide your Drexel credentials, set the environment variable DREXEL_EMAIL to your Drexel email (abc123@drexel.edu) and DREXEL_PASSWORD to the password you use to login to Drexel One. You can follow this guide for Windows, and this guide for MacOS to set environment variables.

You will also need to go to this page and make sure "Authenticator app or hardware token" is the preferred sign-in method. Unfortunately, if you use Microsoft Authenticator as your MFA app you will not be able to run the scraper. You will have to delete the Microsoft Authenticator sign in method and install a different MFA app.

There are two ways to provide MFA for the script to authenticate with. The first is easier if you're looking to run the script manually and quickly. The second is better if you are going to be running the script frequently, or if it needs to be automated.

Authenticate manually

You will authenticate the scraper manually as if you were logging into Drexel One, using a one-time code either from an authenticator app or that is texted to you. After setting the DREXEL_EMAIL and DREXEL_PASSWORD environment variables, run the scraper as explained above, and you will be prompted for your verification code.

Authenticate using a secret key

If you set this up, you will not need to manually enter an authentication code each time you run the scraper.

1. Go to connect.drexel.edu.
2. Click 'Help & Settings', then 'Change MFA settings'.
3. Log in to the Microsoft portal, then click 'Add sign-in method' on the 'Security info' tab.
4. Select 'Authenticator app' for the method, and click 'Add'.
5. Select 'I want to use a different authenticator app', and then 'Next'.
6. Select 'Can't scan image?' when prompted with a QR code, and you should see an Account name and Secret key.
7. Set your DREXEL_MFA_SECRET_KEY environment variable to the given Secret key.
8. Select 'Next', you should be prompted to enter an authentication code.
9. With the secret key environment variable set, run python3 src/totp.py which will generate a one-time code.
10. Enter this code into the Microsoft website, and select 'Next'
11. 'Authenticator app' with TOTP should have been added to the list of available methods.

Now, when you run the scraper as explained above, it should authenticate itself automatically using this secret key.

All Colleges

To scrape all colleges instead of just the one specified in the src/config.py, run the following command:

Mac/Linux

python3 src/main.py --all-colleges
`