Data Licensing

What you can do with the data which you receive depends on the Licenses which are imposed by the Crawlers, the jurisdiction of the actual Crawler, and the laws in your jurisdiction. Websites may contain data which falls under the GDPR: the European privacy laws. Your Tasks fall under the European Data Processing regulations.

When you are not storing this data for a long term or transform the data which breaks GDPR restrictions (for instance, by breaking up texts into single lines), then you are probably fine. For other uses, consult a lawyer. Together with the Open Search Foundation, we try to establish standard legal rules.