Support for multiple referrers

Oct 13, 2025 by talynt.org 88 views

Overview

In this article, we will explore the possibility of allowing uploads from multiple direct URLs, while ensuring that all passed URLs are direct media URLs, on the same site, and from the same referrer. This feature is particularly useful in scenarios where it is not possible to grab URLs server-side, such as when dealing with rate-limited APIs or when working with platforms that restrict server-side access.

Use Cases

There are several use cases where allowing uploads from multiple direct URLs can be beneficial. For instance, when working with social media platforms like Twitter, it may not be possible to grab URLs server-side due to rate limits. In such cases, client-side JavaScript can be used to fetch the URLs, which can then be uploaded to the server. Another example is when dealing with rewards or incentives on platforms like Fantia or Ci-en, where the URLs may not be accessible server-side.

Requirements

To implement this feature, we need to consider the following requirements:

Direct Media URLs: All passed URLs must be direct media URLs, which means they should point to a specific media file, such as an image or a video.
Same Site: The URLs passed must be from the same site, which means they should have the same domain and protocol.
Same Referrer: The URLs passed must be from the same referrer, which means they should have the same origin and referer URL.

Implementation

To implement this feature, we can use a combination of client-side JavaScript and server-side code. Here's a high-level overview of the implementation:

Client-Side JavaScript

On the client-side, we can use JavaScript to fetch the URLs and create a new upload entry. We can use the fetch API to make a GET request to the server, passing the URLs as query parameters.

fetch('/uploads/new', {
  method: 'GET',
  headers: {
    'Content-Type': 'application/x-www-form-urlencoded'
  },
  body: `urls[]=https://pbs.twimg.com/media/GjJRPcmXkAAfPud.jpg:orig&urls[]=https://pbs.twimg.com/media/GjJRPcjWgAA9-eU.jpg:orig&ref=https://twitter.com/mocomori_/status/1887661486169825446`
})
.then(response => response.json())
.then(data => console.log(data))
.catch(error => console.error(error));

Server-Side Code

On the server-side, we can use a programming language like Node.js to create a new upload entry. We can use a library like Express.js to handle the request and create a new upload entry.

const express = require('express');
const app = express();

app.get('/uploads/new', (req, res) => {
  const urls = req.query.urls;
  const referrer = req.query.ref;

  // Validate the URLs and referrer
  if (!urls || !referrer) {
    return res.status(400).send('Invalid request');
  }

  // Create a new upload entry
  const upload = new Upload();
  upload.urls = urls.split('&');
  upload.referrer = referrer;

  // Save the upload entry to the database
  upload.save((err, upload) => {
    if (err) {
      return res.status(500).send('Error creating upload entry');
    }

    res.json(upload);
  });
});

Security Considerations

When implementing this feature, we need to consider security implications. Here are some security considerations to keep in mind:

CSRF Protection: We need to implement CSRF protection to prevent cross-site request forgery attacks.
Input Validation: We need to validate the input URLs and referrer to prevent malicious requests.
Rate Limiting: We need to implement rate limiting to prevent abuse and denial-of-service attacks.

Conclusion

Allowing uploads from multiple direct URLs can be a useful feature in scenarios where it is not possible to grab URLs server-side. By implementing this feature, we can provide a better user experience and improve the overall functionality of our application. However, we need to consider security implications and implement necessary measures to prevent abuse and denial-of-service attacks.

Future Work

In the future, we can improve this feature by adding more functionality, such as:

Support for multiple referrers: We can add support for multiple referrers, allowing users to upload URLs from different sources.
Support for different media types: We can add support for different media types, such as videos and audio files.
Improved security measures: We can improve security measures, such as implementing more robust CSRF protection and input validation.

References

Express.js Documentation
Node.js Documentation
CSRF Protection
Input Validation
Allowing Uploads from Multiple Direct URLs: Q&A =====================================================

Q: What is the purpose of allowing uploads from multiple direct URLs?

A: The purpose of allowing uploads from multiple direct URLs is to provide a way for users to upload multiple media files from different sources, while ensuring that all passed URLs are direct media URLs, on the same site, and from the same referrer.

Q: What are the requirements for allowing uploads from multiple direct URLs?

A: The requirements for allowing uploads from multiple direct URLs are:

Direct Media URLs: All passed URLs must be direct media URLs, which means they should point to a specific media file, such as an image or a video.
Same Site: The URLs passed must be from the same site, which means they should have the same domain and protocol.
Same Referrer: The URLs passed must be from the same referrer, which means they should have the same origin and referer URL.

Q: How do I implement allowing uploads from multiple direct URLs?

A: To implement allowing uploads from multiple direct URLs, you can use a combination of client-side JavaScript and server-side code. Here's a high-level overview of the implementation:

Client-Side JavaScript: Use JavaScript to fetch the URLs and create a new upload entry. You can use the fetch API to make a GET request to the server, passing the URLs as query parameters.
Server-Side Code: Use a programming language like Node.js to create a new upload entry. You can use a library like Express.js to handle the request and create a new upload entry.

Q: What are the security considerations for allowing uploads from multiple direct URLs?

A: When implementing allowing uploads from multiple direct URLs, you need to consider security implications. Here are some security considerations to keep in mind:

CSRF Protection: You need to implement CSRF protection to prevent cross-site request forgery attacks.
Input Validation: You need to validate the input URLs and referrer to prevent malicious requests.
Rate Limiting: You need to implement rate limiting to prevent abuse and denial-of-service attacks.

Q: How do I validate the input URLs and referrer?

A: To validate the input URLs and referrer, you can use a combination of regular expressions and URL parsing libraries. Here's an example of how you can validate the input URLs and referrer using Node.js and the url library:

const url = require('url');

const validateUrl = (urlString) => {
  const parsedUrl = url.parse(urlString);
  if (!parsedUrl.protocol || !parsedUrl.host) {
    return false;
  }
  return true;
};

const validateReferrer = (referrerString) => {
  const parsedReferrer = url.parse(referrerString);
  if (!parsedReferrer.protocol || !parsedReferrer.host) {
    return false;
  }
  return true;
};

Q: How do I implement CSRF protection?

A: To implement CSRF protection, you can use a library like csurf in Node.js. Here's an example of how you can implement CSRF protection using csurf:

const csurf = require('csurf');

const csrfProtection = csurf({
  cookie: {
    httpOnly: true,
    secure: true,
    sameSite: 'strict'
  }
});

app.use(csrfProtection);

Q: How do I implement rate limiting?

A: To implement rate limiting, you can use a library like express-rate-limit in Node.js. Here's an example of how you can implement rate limiting using express-rate-limit:

const rateLimit = require('express-rate-limit');

const limiter = rateLimit({
  windowMs: 15 * 60 * 1000, // 15 minutes
  max: 100 // limit each IP to 100 requests per windowMs
});

app.use(limiter);

Conclusion

Allowing uploads from multiple direct URLs can be a useful feature in scenarios where it is not possible to grab URLs server-side. By implementing this feature, you can provide a better user experience and improve the overall functionality of your application. However, you need to consider security implications and implement necessary measures to prevent abuse and denial-of-service attacks.