Add allowed_domains variable to spider definition

[andresp99999]

The allowed_domains settings allow us to control the domains filtered by the OffsiteMiddleware. If not then it does the previous behavior (extract allowed_domains from start_urls)

[andresp99999]

I did the following changes:

• Add test for previous behavior.
• Add test to check for 'all' value in allowed_domains.
• Refactor code a bit as suggested.
• Refactor line in tests to make it shorter than 80 characters.

[pablohoffman]

"This variable defines" is a bit redundant, if you look at how the other attributes are described.

[pablohoffman]

any sounds better than all to me, for allowing any domain.

But I have another suggestion: what about supporting wildcards?. Using * to denote all domains, and would also allow things like *.scrapinghub.com. This way we'd also keep allowed_domains always a list (single value types add calrity IMO).

[pablohoffman]

I realize this suggestion would involve changes to Scrapy's OffsiteMiddleware, but it would a welcome addition I think, and a relatively easy one. Perhaps it's easier (and more flexible) to support regexes instead of wildcard matching?

[andresp99999]

I finally got time to get to this. I made the following changes to the PR:

• Simplify the documentation as suggested.
• Changed the definition a bit based in Pablo suggestion, it is currently like this:
  • If not set then the previous behavior is kept (use the list of start_urls)
  • If set to an empty list then all domains are allowed (similarly to scrapy behavior when is empty).
  • If set set to a list, use the domains in the list.
• I like the idea of using wildcards, etc, but that would require changes to the scrapy middleware. I think that can be done a little later (it should be a separate PR in scrapy anyway first). I'll take a crack at it some time soon. For now I'd like to keep this PR simple.
• Added the line to the validation schema as suggested.
• Refactor the names of the test spiders to make them more meaningful.
• Squashed all the commits into a single one