Skip to content
This repository was archived by the owner on Aug 19, 2026. It is now read-only.
amirongitPublic archive

About

simple webcrawler using bs4 and requests.

Topics

Resources

Stars

0 stars

Watchers

1 watching

Forks

Latest commit

 

History

182 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

webider is an absurd attempt to aggregate all the domains that can be found on the internet.

- example proxy config
```
{
    "https": "socks5h://host:port",
    "http": "http://host:port"
}
```

- how to
```
# after cloning the project, creating a virtual environment and setting up the db uri in config.json
cd webider/src/
pip install -r ../requirements
python main.py
```

- missing dependency for SOCKS?
```
pip install requests[socks]
```

About

simple webcrawler using bs4 and requests.

Topics

Resources

Stars

0 stars

Watchers

1 watching

Forks

Releases

Used by

Contributors

Languages