Add request body file storage while receiving

README.md

@@ -23,6 +23,7 @@ Crow is a C++ framework for creating HTTP or Websocket web services. It uses rou
  - Middleware support for extensions.
  - HTTP/1.1 and Websocket support.
  - Multi-part request and response support.
+ - Request body files for large uploads (write to disk while receiving).
  - Uses modern C++ (11/14)
 
 ### Still in development

docs/guides/body-file.md

@@ -0,0 +1,60 @@
+<span class="tag">[:octicons-feed-tag-16: master](https://github.com/CrowCpp/Crow)</span>
+
+By default Crow stores a request body in `req.body` (`std::string`). That is
+the right default for JSON and form fields, but a large upload then occupies
+the same amount of RAM as the file.
+
+A route can ask Crow to write the body to a unique file **while the bytes
+arrive**. `req.body` stays empty; the handler uses `req.body_file_path`.
+
+This is the request-side counterpart of returning a file with
+`response.set_static_file_info()`: the payload is never held as one string.
+
+## Route option
+
+```cpp
+CROW_ROUTE(app, "/upload")
+  .methods(crow::HTTPMethod::Put, crow::HTTPMethod::Post)
+  .body_file()([](const crow::request& req) {
+      if (!req.has_body_file())
+          return crow::response(500);
+      // req.body is empty; the bytes are in req.body_file_path
+      std::ifstream in(req.body_file_path, std::ios::binary);
+      // ...
+      return crow::response(200);
+  });
+```
+
+Pass a directory to `.body_file(<directory>)` to override the app default for
+that route. Crow always generates a unique file name, so concurrent requests
+do not share a path.
+
+## App options
+
+```cpp
+app.body_file_directory("uploads");          // default: system temp directory
+app.max_body_file_size(64ull * 1024 * 1024); // default: 0 = unlimited
+```
+
+When the written size would exceed `max_body_file_size()`, Crow discards the
+partial file, still consumes the rest of the request so keep-alive stays in
+sync, and responds `413 Payload Too Large` without running the handler.
+
+## Lifetime
+
+The file is deleted after the response is sent. Call `req.keep_body_file()`
+if the application will use the file after the handler returns (for example
+after renaming it into place). Copy `body_file_path` if another thread will
+open it.
+
+A client that disconnects before the body is complete never reaches the
+handler; the partial file is removed.
+
+## What this is not
+
+`.body_file()` stores the **raw** request body. `multipart/form-data` is still
+parsed from `req.body` by `crow::multipart::message`. Saving individual
+multipart parts to disk as they arrive is a separate feature.
+
+The handler still runs only after the full body has been received. The gain
+is bounded memory, not an incremental handler API.

docs/guides/routes.md

@@ -82,6 +82,7 @@ Handlers can also use information from the request by adding it as a parameter `
 
 You can also access the URL parameters in the handler using `#!cpp req.url_params.get("param_name");`. If the parameter doesn't exist, `nullptr` is returned.<br><br>
 
+To write a large request body to a file while it is received (instead of filling `req.body`), use `#!cpp .body_file()` on the route. See [Request body files](body-file.md).
 
 !!! note "Note &nbsp;&nbsp;&nbsp;&nbsp; <span class="tag">[:octicons-feed-tag-16: master](https://github.com/CrowCpp/Crow)</span>"
 

examples/CMakeLists.txt

@@ -93,6 +93,10 @@ add_executable(example_file_upload example_file_upload.cpp)
 add_warnings_optimizations(example_file_upload)
 target_link_libraries(example_file_upload PUBLIC Crow::Crow)
 
+add_executable(example_body_file example_body_file.cpp)
+add_warnings_optimizations(example_body_file)
+target_link_libraries(example_body_file PUBLIC Crow::Crow)
+
 add_executable(example_unix_socket example_unix_socket.cpp)
 add_warnings_optimizations(example_unix_socket)
 target_link_libraries(example_unix_socket PUBLIC Crow::Crow)

examples/example_body_file.cpp

@@ -0,0 +1,29 @@
+#include "crow.h"
+
+#include <fstream>
+#include <iterator>
+#include <string>
+
+int main()
+{
+    crow::SimpleApp app;
+    app.body_file_directory("uploads").max_body_file_size(64ull * 1024 * 1024);
+
+    // PUT or POST the raw body; Crow writes it to disk as the bytes arrive.
+    CROW_ROUTE(app, "/upload")
+      .methods(crow::HTTPMethod::Put, crow::HTTPMethod::Post)
+      .body_file()([](const crow::request& req) {
+          if (!req.has_body_file())
+              return crow::response(500);
+
+          std::ifstream in(req.body_file_path, std::ios::binary);
+          const std::string contents((std::istreambuf_iterator<char>(in)), std::istreambuf_iterator<char>());
+
+          crow::json::wvalue reply;
+          reply["bytes"] = req.body_file_size;
+          reply["preview"] = contents.substr(0, 32);
+          return crow::response(reply);
+      });
+
+    app.port(18080).multithreaded().run();
+}

include/crow/app.h

@@ -520,6 +520,51 @@ namespace crow
             return res_stream_threshold_;
         }
 
+        /// \brief Directory used for request body files created by routes that call `body_file()`.
+        ///
+        /// Empty (the default) uses the system temporary directory. Unique names are generated per request.
+        self_t& body_file_directory(std::string directory)
+        {
+            body_file_directory_ = std::move(directory);
+            return *this;
+        }
+
+        const std::string& body_file_directory() const
+        {
+            return body_file_directory_;
+        }
+
+        /// \brief Maximum number of bytes written to a request body file (0 = unlimited, the default).
+        ///
+        /// A body that exceeds the limit is discarded and the client receives HTTP 413.
+        /// Remaining body bytes are still consumed so a keep-alive connection stays in sync.
+        self_t& max_body_file_size(uint64_t bytes)
+        {
+            max_body_file_size_ = bytes;
+            return *this;
+        }
+
+        uint64_t max_body_file_size() const
+        {
+            return max_body_file_size_;
+        }
+
+        /// \brief Whether the matched route writes the request body to a file (used by the connection).
+        bool should_save_body_to_file(const routing_handle_result& found)
+        {
+            BaseRule* rule = router_.get_rule(found);
+            return rule && rule->stores_body_in_file();
+        }
+
+        /// \brief Create a unique path for a request body file (used by the connection).
+        std::string create_body_file_path(const routing_handle_result& found)
+        {
+            BaseRule* rule = router_.get_rule(found);
+            std::string directory = body_file_directory_;
+            if (rule && !rule->body_file_directory().empty())
+                directory = rule->body_file_directory();
+            return utility::create_temporary_file(directory);
+        }
 
         self_t& register_blueprint(Blueprint& blueprint)
         {
@@ -920,6 +965,8 @@ namespace crow
         detail::socket::tcp_socket_options tcp_socket_options_{};
         detail::socket::tcp_socket_options websocket_tcp_socket_options_{};
         size_t res_stream_threshold_ = 1048576;
+        std::string body_file_directory_;
+        uint64_t max_body_file_size_{0};
         Router router_;
         bool static_routes_added_{false};
 

include/crow/http_connection.h

@@ -141,6 +141,17 @@ namespace crow
                 parser_.done();
                 need_to_call_after_handlers_ = true;
                 complete_request();
+                return;
+            }
+            if (handler_->should_save_body_to_file(*routing_handle_result_))
+            {
+                const std::string path = handler_->create_body_file_path(*routing_handle_result_);
+                if (!parser_.open_body_file(path, handler_->max_body_file_size()))
+                {
+                    req_.body_error_code = status::INTERNAL_SERVER_ERROR;
+                    parser_.discard_remaining_body();
+                    CROW_LOG_ERROR << "Failed to open request body file";
+                }
             }
         }
 
@@ -160,7 +171,12 @@ namespace crow
             add_keep_alive_ = req_.keep_alive;
             close_connection_ = req_.close_connection;
 
-            if (req_.check_version(1, 1)) // HTTP/1.1
+            if (req_.body_error_code)
+            {
+                is_invalid_request = true;
+                res = response(req_.body_error_code);
+            }
+            else if (req_.check_version(1, 1)) // HTTP/1.1
             {
                 if (!req_.headers.count("host"))
                 {

include/crow/http_request.h

@@ -51,6 +51,9 @@ namespace crow // NOTE: Already documented in "crow/app.h"
         query_string url_params; ///< The parameters associated with the request. (everything after the `?` in the URL)
         ci_map headers;
         std::string body;
+        std::string body_file_path; ///< Path of the file that received the body, if the route used `body_file()`. Empty when the body is in `body`.
+        uint64_t body_file_size{};  ///< Number of bytes written to `body_file_path` when the body was stored on disk.
+        int body_error_code{};      ///< Non-zero when the body could not be stored (`413` or `500`); the handler is not invoked.
         std::string remote_ip_address; ///< The IP address from which the request was sent.
         unsigned char http_ver_major, http_ver_minor;
         bool keep_alive,    ///< Whether or not the server should send a `connection: Keep-Alive` header to the client.
@@ -60,6 +63,7 @@ namespace crow // NOTE: Already documented in "crow/app.h"
         void* middleware_context{};
         void* middleware_container{};
         asio::io_context* io_context{};
+        mutable bool persist_body_file{false}; ///< When true, a body file is not deleted after the request. Set via `keep_body_file()`.
 
         /// Construct an empty request. (sets the method to `GET`)
         request():
@@ -91,12 +95,29 @@ namespace crow // NOTE: Already documented in "crow/app.h"
         /// Get the body as parameters in QS format.
 
         ///
-        /// This is meant to be used with requests of type "application/x-www-form-urlencoded"
+        /// This is meant to be used with requests of type "application/x-www-form-urlencoded".
+        /// It reads `body`; it does not open `body_file_path`.
         const query_string get_body_params() const
         {
             return query_string(body, false);
         }
 
+        /// True when the request body was written to `body_file_path` instead of `body`.
+        bool has_body_file() const
+        {
+            return !body_file_path.empty();
+        }
+
+        /// Keep the body file after the response is sent.
+
+        ///
+        /// Without this call the file is deleted once the connection is done with the request.
+        /// The handler must copy `body_file_path` if it will use the file after returning.
+        void keep_body_file() const
+        {
+            persist_body_file = true;
+        }
+
         /// Send data to whoever made this request with a completion handler and return immediately.
         template<typename CompletionHandler>
         void post(CompletionHandler handler)