diff --git a/README.md b/README.md index 8edcf2909f..a44ebc74a1 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,7 @@ Crow is a C++ framework for creating HTTP or Websocket web services. It uses rou - Middleware support for extensions. - HTTP/1.1 and Websocket support. - Multi-part request and response support. + - Request body size limit (`max_body_size`) and optional body-to-file / body-sink uploads. - Uses modern C++ (11/14) ### Still in development diff --git a/docs/guides/app.md b/docs/guides/app.md index daa17217a2..9225124093 100644 --- a/docs/guides/app.md +++ b/docs/guides/app.md @@ -52,6 +52,25 @@ app.tcp_nodelay(true) .run(); ``` +## Request body size +[:octicons-feed-tag-16: master](https://github.com/CrowCpp/Crow) + +By default Crow accepts a request body of any size. `#!cpp app.max_body_size(bytes)` sets an app-wide limit (`UINT64_MAX` is unlimited). A route can override it with `#!cpp .max_body_size(bytes)`. + +The advertised `Content-Length` is checked when headers complete, before `100 Continue`. Chunked bodies are counted as they arrive. An over-limit request is answered `413 Payload Too Large`; the route handler does not run. Whether the rejection happens on the advertised `Content-Length` alone or partway through a streaming (chunked) body, the connection drains and discards whatever the client still sends rather than closing on unread bytes, which could reset a peer that is still mid-upload; the drain is bounded by the same deadline timer (`app.timeout()`, default 5s) a slow client already gets. The same cap applies to 404, 405, and slash-redirects. + +```cpp +crow::SimpleApp app; +app.max_body_size(64ull * 1024 * 1024); + +CROW_ROUTE(app, "/upload") + .methods(crow::HTTPMethod::Post) + .max_body_size(8ull * 1024 * 1024) + ([](const crow::request& req) { + return crow::response(200); + }); +``` +

For more info on middlewares, check out [this page](middleware.md).

diff --git a/docs/guides/body-file.md b/docs/guides/body-file.md new file mode 100644 index 0000000000..3f6de974f7 --- /dev/null +++ b/docs/guides/body-file.md @@ -0,0 +1,122 @@ +[:octicons-feed-tag-16: master](https://github.com/CrowCpp/Crow) + +By default Crow stores a request body in `req.body` (`std::string`). That is +the right default for JSON and form fields, but a large upload then occupies +the same amount of RAM as the file. + +A route can divert body bytes **while they arrive** to a sink instead, via +`#!cpp .body_sink(factory)`. The built-in sink is a unique file +(`crow::FileBodySink`, an opt-in header — see below); you can also supply +your own (flash, SD, a bounded buffer) when there is no writable filesystem. + +The handler still runs only after the full body has been received. Writes run +synchronously on the connection's io thread, before the `Host` header check, +before middleware, and before `100 Continue`. A sink that throws, or returns +`false` from `write()`/`finish()`, ends the request as a 500 with the +connection closed; the route handler never runs. + +This is the request-side counterpart of returning a file with +`response.set_static_file_info()`: the payload is never held as one string. + +Size limits are `app.max_body_size()` / `.max_body_size()` — see +[Request body size](app.md#request-body-size). Over-limit requests are 413 +and the connection closes; nothing the sink wrote is kept. + +## File convenience + +```cpp +#include "crow.h" +#include "crow/file_body_sink.h" // opt-in: not pulled in by crow.h + +crow::SimpleApp app; +app.max_body_size(64ull * 1024 * 1024); + +CROW_ROUTE(app, "/upload") + .methods(crow::HTTPMethod::Put, crow::HTTPMethod::Post) + .body_sink(crow::FileBodySink::factory("uploads")) // empty: system temp directory + ([](const crow::request& req) { + auto* file = crow::FileBodySink::from(req); + if (!file) + return crow::response(200); // e.g. an empty PUT: see "Empty bodies" below + // req.body is empty; the bytes are in file->path() + std::ifstream in(file->path(), std::ios::binary); + // ... + return crow::response(200); + }); +``` + +`crow::FileBodySink::factory(directory)` resolves `directory` to an absolute +path once, at the point you call it, and throws +`std::filesystem::filesystem_error` if it does not already exist — Crow does +not create it for you, and does not fail silently. Crow always generates a +unique file name (POSIX: a retried `open(O_CREAT | O_EXCL)`; Windows: a +retried `CREATE_NEW`), so concurrent requests never share a path. The +descriptor is kept open until the body is complete; the handler then reads +`file->path()`. + +The file is deleted when the last `crow::request` copy that reached the +handler is destroyed — normally right after the response has been fully +sent. Call `file->keep()` if the application will use the file after that +(for example after renaming it into place); the destructor then only closes +the descriptor. + +A client that disconnects before the body is complete never reaches the +handler; the partial file is removed. Bodyless GET/HEAD (and +`Content-Length: 0`) do not create a file — `crow::FileBodySink::from(req)` +returns `nullptr` in that case, same as for any route where the body didn't +go to this sink. + +`crow::FileBodySink::from()` uses `dynamic_cast` and so needs RTTI; that is +why this header is opt-in and never pulled in by `crow.h` or +`crow/parser.h`. A build with `-fno-rtti` (or asio's `-DASIO_NO_TYPEID`) can +use `crow::BodySink` directly and simply not include +`crow/file_body_sink.h`. + +## Custom sink + +```cpp +struct FlashSink : crow::BodySink { + bool write(const char* data, std::size_t length) override; + bool finish() override; +}; + +CROW_ROUTE(app, "/upload") + .methods(crow::HTTPMethod::Post) + .body_sink([](const crow::request& req) -> std::unique_ptr { + return std::make_unique(req); + }) + ([](const crow::request&) { + return crow::response(200); + }); +``` + +The factory runs once per request, after headers and before the body; `req` +carries the method, url, headers and `remote_ip_address` (already set, so a +factory can apply a per-peer policy), but not `body` or `body_sink` yet. +Returning `nullptr` from the factory means "no sink for this request" — the +body goes to `req.body` as usual, it is not an error. Throwing from the +factory, or from `write()`/`finish()`, is a 500. + +Calling `.body_sink(...)` again on the same route replaces the factory +(last-call-wins). + +## Empty bodies + +A `.body_sink(...)` route only opens a sink when a body is actually coming +(`Content-Length` > 0, or chunked framing); a bodyless request never calls +the factory, so `req.body_sink` (and `crow::FileBodySink::from(req)`) is +`nullptr`. Handle that case explicitly — see the example above. Note that an +empty **chunked** body (`0\r\n\r\n`) still counts as "a body is coming" and +does open the sink (into an empty file, for the built-in one), unlike +`Content-Length: 0`. + +## What this is not + +`.body_sink(...)` stores the **raw** request body. `multipart/form-data` is +parsed from `req.body` by `crow::multipart::message`; a route with a sink +diverts every body regardless of `Content-Type`, so `req.body` is empty and +`crow::multipart::message` cannot be used on it. Saving individual multipart +parts as they arrive is a separate feature. + +The route handler is not invoked per chunk. Incremental processing belongs in +`BodySink::write`. diff --git a/docs/guides/routes.md b/docs/guides/routes.md index 4552905fac..ad9dc11a71 100644 --- a/docs/guides/routes.md +++ b/docs/guides/routes.md @@ -82,6 +82,9 @@ Handlers can also use information from the request by adding it as a parameter ` You can also access the URL parameters in the handler using `#!cpp req.url_params.get("param_name");`. If the parameter doesn't exist, `nullptr` is returned.

+To cap the request body for one route, use `#!cpp .max_body_size(bytes)` (see [Request body size](app.md#request-body-size)). Without a per-route value, `#!cpp app.max_body_size()` applies, including on 404 and 405. + +To write a large request body to a file (or a custom sink) while it is received instead of filling `req.body`, use `#!cpp .body_sink(...)`. See [Request body files](body-file.md). !!! note "Note      [:octicons-feed-tag-16: master](https://github.com/CrowCpp/Crow)" diff --git a/examples/CMakeLists.txt b/examples/CMakeLists.txt index f564ae5323..0d38aca4ad 100644 --- a/examples/CMakeLists.txt +++ b/examples/CMakeLists.txt @@ -93,6 +93,10 @@ add_executable(example_file_upload example_file_upload.cpp) add_warnings_optimizations(example_file_upload) target_link_libraries(example_file_upload PUBLIC Crow::Crow) +add_executable(example_body_file example_body_file.cpp) +add_warnings_optimizations(example_body_file) +target_link_libraries(example_body_file PUBLIC Crow::Crow) + add_executable(example_unix_socket example_unix_socket.cpp) add_warnings_optimizations(example_unix_socket) target_link_libraries(example_unix_socket PUBLIC Crow::Crow) diff --git a/examples/example_body_file.cpp b/examples/example_body_file.cpp new file mode 100644 index 0000000000..59e31c7589 --- /dev/null +++ b/examples/example_body_file.cpp @@ -0,0 +1,38 @@ +#include "crow.h" +#include "crow/file_body_sink.h" + +#include +#include +#include + +int main() +{ + crow::SimpleApp app; + app.max_body_size(64ull * 1024 * 1024); + + // PUT or POST the raw body; Crow writes it to disk as the bytes arrive. + CROW_ROUTE(app, "/upload") + .methods(crow::HTTPMethod::Put, crow::HTTPMethod::Post) + .body_sink(crow::FileBodySink::factory("uploads")) // requires ./uploads to exist + ([](const crow::request& req) { + auto* file = crow::FileBodySink::from(req); + if (!file) + { + // No incoming body (e.g. an empty PUT): the sink never opened. + crow::json::wvalue reply; + reply["bytes"] = 0; + reply["preview"] = ""; + return crow::response(reply); + } + + std::ifstream in(file->path(), std::ios::binary); + const std::string contents((std::istreambuf_iterator(in)), std::istreambuf_iterator()); + + crow::json::wvalue reply; + reply["bytes"] = contents.size(); + reply["preview"] = contents.substr(0, 32); + return crow::response(reply); + }); + + app.port(18080).multithreaded().run(); +} diff --git a/include/crow.h b/include/crow.h index cb3720cff3..bd07f52baf 100644 --- a/include/crow.h +++ b/include/crow.h @@ -13,6 +13,7 @@ #include "crow/utility.h" #include "crow/common.h" #include "crow/http_request.h" +#include "crow/body_sink.h" #include "crow/websocket.h" #include "crow/parser.h" #include "crow/http_response.h" diff --git a/include/crow/app.h b/include/crow/app.h index 7aa60522db..8dec28545b 100644 --- a/include/crow/app.h +++ b/include/crow/app.h @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -349,6 +350,30 @@ namespace crow return max_payload_; } + /// \brief Set the maximum request body size in bytes (`UINT64_MAX` = unlimited, the default). + /// + /// Applied to every request, including 404, 405, and slash-redirects. + /// Advertised `Content-Length` is checked at headers-complete, before `100 Continue`. + /// Chunked bodies are counted as they arrive. An over-limit request is answered 413 + /// and the connection is closed without reading (or draining) the body. + self_t& max_body_size(uint64_t bytes) + { + max_body_size_ = bytes; + return *this; + } + + /// \brief Get the app-wide request body size limit + uint64_t max_body_size() const + { + return max_body_size_; + } + + /// \brief Effective body size limit for a routing result (used by the connection). + uint64_t effective_max_body_size(const routing_handle_result& found) const + { + return router_.effective_max_body_size(found, max_body_size_); + } + self_t& signal_clear() { signals_.clear(); @@ -520,6 +545,16 @@ namespace crow return res_stream_threshold_; } + /// \brief Create a body sink for the matched route, or nullptr for in-memory `req.body`. + std::unique_ptr make_body_sink(const routing_handle_result& found, const request& req) const + { + return router_.make_body_sink(found, req); + } + + bool uses_body_sink(const routing_handle_result& found) const + { + return router_.uses_body_sink(found); + } self_t& register_blueprint(Blueprint& blueprint) { @@ -914,6 +949,7 @@ namespace crow unsigned int concurrency_ = 2; std::atomic_bool is_bound_ = false; uint64_t max_payload_{UINT64_MAX}; + uint64_t max_body_size_{UINT64_MAX}; std::string server_name_ = std::string("Crow/") + VERSION; std::string bindaddr_ = "0.0.0.0"; bool use_unix_ = false; diff --git a/include/crow/body_sink.h b/include/crow/body_sink.h new file mode 100644 index 0000000000..366de441ba --- /dev/null +++ b/include/crow/body_sink.h @@ -0,0 +1,29 @@ +#pragma once + +#include +#include +#include + +namespace crow +{ + struct request; + + /// Destination for request body bytes as they arrive from the parser. + /// + /// `write()`/`finish()` run on the connection's io thread, at headers-complete, + /// before the `Host` check, before middleware, and before `100 Continue`. A + /// thrown exception or a `false` return is a 500 with the connection closed; + /// the route handler never runs. If `finish()` was never called, the body did + /// not arrive in full (client disconnect, over-limit body, or a write failure). + struct BodySink + { + virtual ~BodySink() = default; + virtual bool write(const char* data, std::size_t length) = 0; + virtual bool finish() = 0; + }; + + /// `req` carries method, url, headers and `remote_ip_address` (set before the + /// factory runs); `req.body` and `req.body_sink` are not yet populated. + /// Returning `nullptr` keeps the body in `req.body` instead of diverting it. + using BodySinkFactory = std::function(const request&)>; +} // namespace crow diff --git a/include/crow/common.h b/include/crow/common.h index a8e58f4abc..aaf395fb27 100644 --- a/include/crow/common.h +++ b/include/crow/common.h @@ -288,10 +288,10 @@ namespace crow struct routing_handle_result { bool catch_all{false}; - size_t rule_index; + size_t rule_index{0}; std::vector blueprint_indices; routing_params r_params; - HTTPMethod method; + HTTPMethod method{}; routing_handle_result() {} diff --git a/include/crow/file_body_sink.h b/include/crow/file_body_sink.h new file mode 100644 index 0000000000..35f52a5cda --- /dev/null +++ b/include/crow/file_body_sink.h @@ -0,0 +1,222 @@ +#pragma once + +// Opt-in: not included by "crow.h" or "crow/parser.h". A build that has no +// writable filesystem, or that disables RTTI (`-fno-rtti`, `-DASIO_NO_TYPEID`), +// can use `crow::BodySink` directly and never pull this header in. + +#include +#include +#include +#include +#include +#include +#include +#include + +#ifndef _WIN32 +#include +#include +#include +#else +#ifndef WIN32_LEAN_AND_MEAN +#define WIN32_LEAN_AND_MEAN +#endif +#include +#endif + +#include "crow/body_sink.h" +#include "crow/http_request.h" +#include "crow/utility.h" + +namespace crow +{ + /// File-backed `BodySink`. Give it to `.body_sink(...)` via `factory()`; read + /// the result back from the handler with `from()`. + /// + /// The descriptor is created `O_CREAT|O_EXCL|O_WRONLY` (Windows: `CREATE_NEW`), + /// mode 0600, and kept open until `finish()`. The file is unlinked when the + /// sink is destroyed, unless `keep()` was called; since `request::body_sink` + /// is a `shared_ptr`, that is when the last copy of the request goes away. + class FileBodySink : public BodySink + { + public: + /// A `BodySinkFactory` that writes each request's body to a uniquely + /// named file under `directory`. `directory` is resolved to an absolute + /// path once, here, and must already exist; empty uses the system + /// temporary directory. Throws `std::filesystem::filesystem_error` if it + /// cannot be resolved — call this while setting up the route, not per + /// request, so a bad directory fails loudly at startup. + static BodySinkFactory factory(std::string directory = {}) + { + auto resolved = std::make_shared(resolve_directory(directory)); + return [resolved](const request&) -> std::unique_ptr { + return create(*resolved); + }; + } + + /// The `FileBodySink` behind `req.body_sink`, or `nullptr` if the route + /// used a different sink (or none). + static FileBodySink* from(const request& req) + { + return dynamic_cast(req.body_sink.get()); + } + + FileBodySink(const FileBodySink&) = delete; + FileBodySink& operator=(const FileBodySink&) = delete; + + ~FileBodySink() override + { + close_handle(); + if (!keep_ && !path_.empty()) + { + std::error_code ec; + std::filesystem::remove(path_, ec); + } + } + + bool write(const char* data, std::size_t length) override + { +#ifndef _WIN32 + if (fd_ < 0) + return false; + std::size_t written = 0; + while (written < length) + { + const ssize_t n = ::write(fd_, data + written, length - written); + if (n < 0) + { + if (errno == EINTR) + continue; + return false; + } + if (n == 0) + return false; + written += static_cast(n); + } + return true; +#else + if (handle_ == INVALID_HANDLE_VALUE) + return false; + std::size_t written = 0; + while (written < length) + { + DWORD n = 0; + // static_cast(MAXDWORD) is still exact on 32-bit size_t + // (both are 32-bit there), so this chunking is correct on 32- + // and 64-bit Windows alike. + const DWORD chunk = static_cast( + std::min(length - written, static_cast(MAXDWORD))); + if (!WriteFile(handle_, data + written, chunk, &n, nullptr) || n == 0) + return false; + written += n; + } + return true; +#endif + } + + bool finish() override + { + return close_handle(); + } + + /// The path of the file, valid once the factory has created it. + const std::string& path() const + { + return path_; + } + + /// Do not delete the file when this sink is destroyed (i.e. when the + /// last copy of the request that reached the handler goes away). + void keep() + { + keep_ = true; + } + + private: + static std::string resolve_directory(const std::string& directory) + { + std::error_code ec; + std::filesystem::path dir = directory.empty() ? + std::filesystem::temp_directory_path(ec) : + std::filesystem::absolute(directory, ec); + if (ec) + throw std::filesystem::filesystem_error("body_file directory unavailable", directory, ec); + if (!std::filesystem::is_directory(dir, ec) || ec) + throw std::filesystem::filesystem_error( + "body_file directory does not exist", dir, + std::make_error_code(std::errc::no_such_file_or_directory)); + return dir.string(); + } + + static std::unique_ptr create(const std::string& directory) + { +#ifndef _WIN32 + const std::filesystem::path dir(directory); + for (int attempt = 0; attempt < 128; ++attempt) + { + const std::string name = "crow-body-" + utility::random_alphanum(16); + const std::string path = (dir / name).string(); + const int fd = ::open(path.c_str(), O_CREAT | O_EXCL | O_WRONLY | O_CLOEXEC, 0600); + if (fd >= 0) + return std::unique_ptr(new FileBodySink(path, fd)); + if (errno != EEXIST) + throw std::system_error(errno, std::generic_category(), "crow::FileBodySink: open"); + } + throw std::runtime_error("crow::FileBodySink: could not find a unique file name"); +#else + const std::filesystem::path dir(directory); + for (int attempt = 0; attempt < 128; ++attempt) + { + const std::string name = "crow-body-" + utility::random_alphanum(16); + const auto path = dir / name; + SECURITY_ATTRIBUTES sa{}; + sa.nLength = sizeof(sa); + sa.bInheritHandle = FALSE; + HANDLE handle = CreateFileW(path.wstring().c_str(), GENERIC_WRITE, 0, &sa, CREATE_NEW, + FILE_ATTRIBUTE_NORMAL, nullptr); + if (handle != INVALID_HANDLE_VALUE) + return std::unique_ptr(new FileBodySink(path.string(), handle)); + if (GetLastError() != ERROR_FILE_EXISTS) + throw std::system_error(static_cast(GetLastError()), std::system_category(), + "crow::FileBodySink: CreateFileW"); + } + throw std::runtime_error("crow::FileBodySink: could not find a unique file name"); +#endif + } + +#ifndef _WIN32 + FileBodySink(std::string path, int fd): + path_(std::move(path)), fd_(fd) + {} +#else + FileBodySink(std::string path, HANDLE handle): + path_(std::move(path)), handle_(handle) + {} +#endif + + bool close_handle() + { +#ifndef _WIN32 + if (fd_ < 0) + return true; + const int fd = fd_; + fd_ = -1; + return ::close(fd) == 0; +#else + if (handle_ == INVALID_HANDLE_VALUE) + return true; + const HANDLE handle = handle_; + handle_ = INVALID_HANDLE_VALUE; + return CloseHandle(handle) != 0; +#endif + } + + std::string path_; + bool keep_{false}; +#ifndef _WIN32 + int fd_{-1}; +#else + HANDLE handle_{INVALID_HANDLE_VALUE}; +#endif + }; +} // namespace crow diff --git a/include/crow/http_connection.h b/include/crow/http_connection.h index 2ebf72a956..379281911e 100644 --- a/include/crow/http_connection.h +++ b/include/crow/http_connection.h @@ -121,8 +121,36 @@ namespace crow } } - void handle_header() + /// Apply `max_body_size` before 100-continue. Return 1 to skip the body (F_SKIPBODY). + int handle_header() { + body_error_status_ = 0; + // Set before a body_sink factory can run, so a sink can apply a per-peer policy. + req_.remote_ip_address = adaptor_.address(); + const uint64_t limit = handler_->effective_max_body_size(*routing_handle_result_); + parser_.set_max_body_size(limit); + + if (parser_.content_length != CROW_ULLONG_MAX && + limit != UINT64_MAX && parser_.content_length > limit) + { + body_error_status_ = status::PAYLOAD_TOO_LARGE; + return 1; + } + + const bool use_body_sink = parser_.has_incoming_body() && + handler_->uses_body_sink(*routing_handle_result_); + if (use_body_sink) + { + try + { + parser_.open_body_sink(handler_->make_body_sink(*routing_handle_result_, req_)); + } + catch (...) + { + body_error_status_ = status::INTERNAL_SERVER_ERROR; + return 1; + } + } // HTTP 1.1 Expect: 100-continue if (req_.http_ver_major == 1 && req_.http_ver_minor == 1 && get_header_value(req_.headers, "expect") == "100-continue") { @@ -142,6 +170,19 @@ namespace crow need_to_call_after_handlers_ = true; complete_request(); } + return 0; + } + + void reject_body(int code) + { + if (body_error_status_ == 0) + body_error_status_ = code; + handle(); + } + + bool parser_should_abort() const + { + return body_error_status_ != 0; } void handle() @@ -156,10 +197,25 @@ namespace crow req_.middleware_context = static_cast(&ctx_); req_.middleware_container = static_cast(middlewares_); req_.io_context = &adaptor_.get_io_context(); - req_.remote_ip_address = adaptor_.address(); add_keep_alive_ = req_.keep_alive; close_connection_ = req_.close_connection; + if (body_error_status_) + { + req_.body.clear(); + res = response(body_error_status_); + // Explicit, rather than relying on response::operator=(&&) happening to + // omit skip_body: a HEAD request must never get a body, regardless of + // which error status (413, or 500 from a body_sink failure) sent it here. + res.skip_body = (req_.method == HTTPMethod::Head); + res.set_header("Connection", "close"); + res.end(); + close_connection_ = true; + add_keep_alive_ = false; + need_to_call_after_handlers_ = true; + complete_request(); + return; + } if (req_.check_version(1, 1)) // HTTP/1.1 { if (!req_.headers.count("host")) @@ -309,7 +365,7 @@ namespace crow while (is.gcount() > 0) { buffers[0] = asio::buffer(buf, is.gcount()); - error_code ec = do_write_sync(buffers); + error_code ec = do_write_sync(buffers, /*clear_parser=*/false); if (ec) { CROW_LOG_ERROR << ec << " - buffer write error happened while sending content of file " << res.file_info.path << ". Writing stopped premature."; @@ -318,7 +374,10 @@ namespace crow is.read(buf, sizeof(buf)); } } - if (close_connection_) + // A body-error response leaves the read side open for do_read()'s + // linger_close() to drain instead of closing here, so a peer still + // mid-upload isn't RST'd before it reads the response. + if (close_connection_ && !body_error_status_) { adaptor_.shutdown_readwrite(); adaptor_.close(); @@ -366,7 +425,7 @@ namespace crow { size_t to_transfer = CROW_MIN(16384UL, length - transferred); buffers[0] = asio::const_buffer(data + transferred, to_transfer); - ec = do_write_sync(buffers); + ec = do_write_sync(buffers, /*clear_parser=*/false); if (ec) { CROW_LOG_ERROR << ec << " - " << transferred << " - buffer write error happened while sending response. Writing stopped premature."; break; @@ -374,7 +433,9 @@ namespace crow transferred += to_transfer; } } - if (close_connection_) + // See the matching comment in do_write_static(): leave the + // close to do_read()'s linger_close() on a body error. + if (close_connection_ && !body_error_status_) { adaptor_.shutdown_readwrite(); adaptor_.close(); @@ -408,9 +469,21 @@ namespace crow { self->cancel_deadline_timer(); self->parser_.done(); - self->adaptor_.shutdown_read(); - self->adaptor_.close(); CROW_LOG_DEBUG << self << " from read(1) with description: \"" << http_errno_description(static_cast(self->parser_.http_errno)) << '\"'; + if (self->body_error_status_) + { + // The rejection response (413, or a sink write/finish + // failure's 500) is already written; shut down the write + // side and drain whatever the client still sends instead + // of closing on unread bytes, which can RST a peer that + // is still mid-upload before it reads the response. + self->linger_close(); + } + else + { + self->adaptor_.shutdown_read(); + self->adaptor_.close(); + } } else if (self->close_connection_) { @@ -431,6 +504,36 @@ namespace crow }); } + /// Shut down the write side (FIN, not RST) after a rejection response and + /// discard whatever the client still sends, instead of closing on unread + /// bytes. Bounded by the same deadline timer a slow client already gets. + void linger_close() + { + adaptor_.shutdown_write(); + start_deadline(); + do_linger_read(); + } + + // Reuses the normal read path's buffer_; safe here because parsing has + // already aborted, so nothing else reads from or writes to it. + void do_linger_read() + { + auto self = this->shared_from_this(); + adaptor_.socket().async_read_some( + asio::buffer(buffer_), + [self](const error_code& ec, std::size_t /*bytes_transferred*/) { + if (!ec && self->adaptor_.is_open()) + { + self->do_linger_read(); + } + else + { + self->cancel_deadline_timer(); + self->adaptor_.close(); + } + }); + } + void do_write() { auto self = this->shared_from_this(); @@ -464,7 +567,12 @@ namespace crow }); } - inline error_code do_write_sync(std::vector& buffers) + /// `clear_parser` must stay false for every call but the last one writing + /// a single response, static file, or streamed body: clearing resets + /// `req` (dropping the connection's `body_sink` reference, e.g. deleting + /// a `FileBodySink`'s file) and must not happen until that whole response + /// has been written, not after its first chunk. + inline error_code do_write_sync(std::vector& buffers, bool clear_parser = true) { error_code ec; asio::write(adaptor_.socket(), buffers, ec); @@ -480,7 +588,7 @@ namespace crow { this->continue_requested = false; } - else + else if (clear_parser) { this->parser_.clear(); } @@ -536,6 +644,7 @@ namespace crow bool need_to_call_after_handlers_{}; bool need_to_start_read_after_complete_{}; bool add_keep_alive_{}; + int body_error_status_{0}; std::tuple* middlewares_; detail::context ctx_; diff --git a/include/crow/http_request.h b/include/crow/http_request.h index ab526f3eef..0874f680ae 100644 --- a/include/crow/http_request.h +++ b/include/crow/http_request.h @@ -10,6 +10,7 @@ #endif #include +#include #include "crow/common.h" #include "crow/ci_map.h" @@ -21,6 +22,8 @@ namespace crow // NOTE: Already documented in "crow/app.h" namespace asio = boost::asio; #endif + struct BodySink; + /// Remove CR (\r) and LF (\n) characters from a header name or value to prevent header injection. inline void sanitize_header_value(std::string& s) { @@ -51,6 +54,11 @@ namespace crow // NOTE: Already documented in "crow/app.h" query_string url_params; ///< The parameters associated with the request. (everything after the `?` in the URL) ci_map headers; std::string body; + /// The sink the route diverted the body to (`.body_sink(...)`), if any. Set + /// before the handler runs; every copy of the request shares it, so it (and + /// whatever it owns, e.g. a `FileBodySink`'s file) lives as long as the last + /// copy. `nullptr` when the body is in `body` instead. + std::shared_ptr body_sink; std::string remote_ip_address; ///< The IP address from which the request was sent. unsigned char http_ver_major, http_ver_minor; bool keep_alive, ///< Whether or not the server should send a `connection: Keep-Alive` header to the client. @@ -91,7 +99,8 @@ namespace crow // NOTE: Already documented in "crow/app.h" /// Get the body as parameters in QS format. /// - /// This is meant to be used with requests of type "application/x-www-form-urlencoded" + /// This is meant to be used with requests of type "application/x-www-form-urlencoded". + /// It reads `body`; a route with `.body_sink(...)` leaves it empty. const query_string get_body_params() const { return query_string(body, false); diff --git a/include/crow/http_response.h b/include/crow/http_response.h index 71a9e0c7a9..61cd83625e 100644 --- a/include/crow/http_response.h +++ b/include/crow/http_response.h @@ -412,7 +412,7 @@ namespace crow auto& status = statusCodes.find(code)->second; buffers.emplace_back(status.data(), status.size()); - if (code >= 400 && body.empty()) + if (code >= 400 && body.empty() && !skip_body) body = statusCodes[code].substr(9); for (auto& kv : headers) diff --git a/include/crow/parser.h b/include/crow/parser.h index 1417d6cb60..6d40bb445b 100644 --- a/include/crow/parser.h +++ b/include/crow/parser.h @@ -1,9 +1,13 @@ #pragma once +#include +#include +#include #include #include -#include +#include "crow/body_sink.h" +#include "crow/common.h" #include "crow/http_request.h" #include "crow/http_parser_merged.h" @@ -82,13 +86,40 @@ namespace crow self->set_connection_parameters(); - self->process_header(); - return 0; + return self->process_header(); } static int on_body(http_parser* self_, const char* at, size_t length) { HTTPParser* self = static_cast(self_); - self->req.body.insert(self->req.body.end(), at, at + length); + if (self->max_body_size_ != UINT64_MAX && + (length > self->max_body_size_ || + self->body_bytes_ > self->max_body_size_ - length)) + { + self->handler_->reject_body(status::PAYLOAD_TOO_LARGE); + return 1; + } + if (self->req.body_sink) + { + bool ok = false; + try + { + ok = self->req.body_sink->write(at, length); + } + catch (...) + { + ok = false; + } + if (!ok) + { + self->handler_->reject_body(status::INTERNAL_SERVER_ERROR); + return 1; + } + } + else + { + self->req.body.insert(self->req.body.end(), at, at + length); + } + self->body_bytes_ += length; return 0; } static int on_message_complete(http_parser* self_) @@ -96,8 +127,26 @@ namespace crow HTTPParser* self = static_cast(self_); self->message_complete = true; + if (self->req.body_sink) + { + bool ok = false; + try + { + ok = self->req.body_sink->finish(); + } + catch (...) + { + ok = false; + } + if (!ok) + { + self->handler_->reject_body(status::INTERNAL_SERVER_ERROR); + return 1; + } + } self->process_message(); - return 0; + // Stop so leftover skipped-body bytes are not parsed as the next request. + return self->handler_->parser_should_abort() ? 1 : 0; } HTTPParser(Handler* handler): http_parser(), @@ -145,17 +194,38 @@ namespace crow header_building_state = 0; qs_point = 0; message_complete = false; + body_bytes_ = 0; + max_body_size_ = UINT64_MAX; state = CROW_NEW_MESSAGE(); } + void set_max_body_size(uint64_t bytes) + { + max_body_size_ = bytes; + } + + /// `sink` may be null: the factory declining means "keep the body in + /// `req.body`", not an error. A factory that wants a 500 should throw. + void open_body_sink(std::unique_ptr sink) + { + req.body_sink = std::move(sink); + } + + bool has_incoming_body() const + { + if (flags & F_CHUNKED) + return true; + return content_length != CROW_ULLONG_MAX && content_length > 0; + } + inline void process_url() { handler_->handle_url(); } - inline void process_header() + inline int process_header() { - handler_->handle_header(); + return handler_->handle_header(); } inline void process_message() @@ -190,6 +260,8 @@ namespace crow private: int header_building_state = 0; bool message_complete = false; + uint64_t body_bytes_{0}; + uint64_t max_body_size_{UINT64_MAX}; std::string header_field; std::string header_value; diff --git a/include/crow/routing.h b/include/crow/routing.h index 42ac0a78f9..47eac80980 100644 --- a/include/crow/routing.h +++ b/include/crow/routing.h @@ -14,6 +14,7 @@ #include "crow/common.h" #include "crow/http_response.h" #include "crow/http_request.h" +#include "crow/body_sink.h" #include "crow/utility.h" #include "crow/logging.h" #include "crow/exceptions.h" @@ -160,6 +161,9 @@ namespace crow // NOTE: Already documented in "crow/app.h" std::string rule_; std::string name_; bool added_{false}; + uint64_t max_body_size_{UINT64_MAX}; + bool max_body_size_override_{false}; + BodySinkFactory body_sink_factory_; std::unique_ptr rule_to_upgrade_; @@ -618,6 +622,28 @@ namespace crow // NOTE: Already documented in "crow/app.h" static_cast(this)->mw_indices_.template push(); return static_cast(*this); } + + /// Override the app-wide request body size limit for this route. + self_t& max_body_size(uint64_t bytes) + { + static_cast(this)->max_body_size_ = bytes; + static_cast(this)->max_body_size_override_ = true; + return static_cast(*this); + } + + /// Write the request body to a sink while it is received, instead of + /// filling `req.body`. + /// + /// The factory runs after headers, before the body; see `BodySinkFactory`. + /// The route handler still runs only after the full body has been + /// received. Calling this again (last-call-wins) replaces the factory. + /// `crow::FileBodySink::factory(directory)` (`crow/file_body_sink.h`, + /// not pulled in by this header) is the built-in file-backed sink. + self_t& body_sink(BodySinkFactory factory) + { + static_cast(this)->body_sink_factory_ = std::move(factory); + return static_cast(*this); + } }; /// A rule that can change its parameters during runtime. @@ -1842,6 +1868,46 @@ namespace crow // NOTE: Already documented in "crow/app.h" return blueprints_; } + /// The rule a `handle_initial()` result matched, or `nullptr` for 404, + /// 405, a slash-redirect, or an unmatched `OPTIONS`. + const BaseRule* matched_rule(const routing_handle_result& found) const + { + if (found.catch_all || found.rule_index <= RULE_SPECIAL_REDIRECT_SLASH) + return nullptr; + if (found.method >= HTTPMethod::InternalMethodCount) + return nullptr; + const auto& rules = per_methods_[static_cast(found.method)].rules; + if (found.rule_index >= rules.size()) + return nullptr; + return rules[found.rule_index]; + } + + /// Effective request body limit for a `handle_initial()` result. + /// + /// 404, 405, slash-redirect, and unmatched OPTIONS use `app_default`. + /// A matched rule uses its override when set, otherwise `app_default`. + uint64_t effective_max_body_size(const routing_handle_result& found, uint64_t app_default) const + { + const BaseRule* rule = matched_rule(found); + if (!rule || !rule->max_body_size_override_) + return app_default; + return rule->max_body_size_; + } + + bool uses_body_sink(const routing_handle_result& found) const + { + const BaseRule* rule = matched_rule(found); + return rule && static_cast(rule->body_sink_factory_); + } + + std::unique_ptr make_body_sink(const routing_handle_result& found, const request& req) const + { + const BaseRule* rule = matched_rule(found); + if (!rule || !rule->body_sink_factory_) + return nullptr; + return rule->body_sink_factory_(req); + } + std::function& exception_handler() { return exception_handler_; diff --git a/mkdocs.yml b/mkdocs.yml index e5013dbb99..eae6f97c38 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -70,6 +70,7 @@ nav: - SBOM Generation: guides/sbom.md - SSL: guides/ssl.md - Static Files: guides/static.md + - Request body files: guides/body-file.md - Blueprints: guides/blueprints.md - Compression: guides/compression.md - Websockets: guides/websockets.md diff --git a/scripts/merge_all.py b/scripts/merge_all.py index 9fc7871c60..c5597ec31d 100755 --- a/scripts/merge_all.py +++ b/scripts/merge_all.py @@ -44,10 +44,16 @@ middlewares_actual = middlewares print("Middlewares: " + str(middlewares_actual)) +# Headers that are opt-in and must not be pulled into crow_all.h: they impose +# a requirement (e.g. RTTI) that the rest of the library doesn't need, so a +# build that can't meet it can still use the amalgamated header. +opt_in_headers = {'file_body_sink.h'} + re_depends = re.compile('^#include \"(.*)\"\n', re.MULTILINE) re_pragma = re.compile('^(.*)#pragma once(.*)\n', re.MULTILINE) headers = [x.rsplit(sep, 1)[-1] for x in glob(pt.join(header_path, '*.h*'))] -headers += ['crow'+sep + x.rsplit(sep, 1)[-1] for x in glob(pt.join(header_path, 'crow'+sep+'*.h*'))] +headers += ['crow'+sep + x.rsplit(sep, 1)[-1] for x in glob(pt.join(header_path, 'crow'+sep+'*.h*')) + if x.rsplit(sep, 1)[-1] not in opt_in_headers] headers += [('crow'+sep+'middlewares'+sep + x + '.h') for x in middlewares_actual] print(headers) edges = defaultdict(list) diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 00db73a7be..6d8e237386 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -22,6 +22,8 @@ enable_testing() # list the test sources set(TEST_SRCS unittest.cpp + max_body_size_tests.cpp + body_file_tests.cpp query_string_tests.cpp unit_tests/test_http_response.cpp unit_tests/test_json.cpp @@ -33,6 +35,7 @@ set(TEST_SRCS add_executable(unittest ${TEST_SRCS}) target_link_libraries(unittest Crow::Crow Catch2::Catch2WithMain) +target_compile_definitions(unittest PRIVATE CROW_TEST_REPO_DIR="${CMAKE_SOURCE_DIR}") add_warnings_optimizations(unittest) add_sanitizer_flags(unittest) diff --git a/tests/body_file_tests.cpp b/tests/body_file_tests.cpp new file mode 100644 index 0000000000..d881193eff --- /dev/null +++ b/tests/body_file_tests.cpp @@ -0,0 +1,1053 @@ +#define CROW_ENABLE_DEBUG +#define CROW_LOG_LEVEL 0 + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#ifndef _WIN32 +#include +#endif + +#include "catch2/catch_all.hpp" +#include "crow.h" +#include "crow/file_body_sink.h" + +using namespace crow; + +#ifdef CROW_USE_BOOST +namespace asio = boost::asio; +using asio_error_code = boost::system::error_code; +#else +using asio_error_code = asio::error_code; +#endif + +#define LOCALHOST_ADDRESS "127.0.0.1" + +#include "http_test_utils.h" +using namespace crow_test_utils; + +namespace +{ + std::string read_all(const std::string& path) + { + std::ifstream in(path, std::ios::binary); + return std::string((std::istreambuf_iterator(in)), std::istreambuf_iterator()); + } + + bool directory_is_empty(const std::filesystem::path& path) + { + std::error_code ec; + const auto it = std::filesystem::directory_iterator(path, ec); + if (ec) + return false; + return it == std::filesystem::directory_iterator(); + } + + bool no_crow_body_files(const std::filesystem::path& path) + { + std::error_code ec; + for (const auto& entry : std::filesystem::recursive_directory_iterator(path, ec)) + { + if (ec) + return false; + if (entry.is_regular_file() && + entry.path().filename().string().rfind("crow-body-", 0) == 0) + return false; + } + return true; + } + + struct TempDir + { + std::filesystem::path path; + + TempDir(): + path(std::filesystem::temp_directory_path() / ("crow-body-file-tests-" + utility::random_alphanum(12))) + { + std::filesystem::create_directories(path); + } + + ~TempDir() + { + std::error_code ec; + std::filesystem::remove_all(path, ec); + } + }; + + struct VectorSink : BodySink + { + std::shared_ptr buf; + bool write(const char* data, std::size_t length) override + { + buf->append(data, length); + return true; + } + bool finish() override { return true; } + }; + + struct FailingWriteSink : BodySink + { + bool write(const char*, std::size_t) override { return false; } + bool finish() override { return true; } + }; + + struct FailingFinishSink : BodySink + { + bool write(const char*, std::size_t) override { return true; } + bool finish() override { return false; } + }; + + struct ThrowingWriteSink : BodySink + { + bool write(const char*, std::size_t) override { throw std::runtime_error("write blew up"); } + bool finish() override { return true; } + }; + + struct ThrowingFinishSink : BodySink + { + bool write(const char*, std::size_t) override { return true; } + bool finish() override { throw std::runtime_error("finish blew up"); } + }; + + // Throws something that is not a std::exception, to exercise the sink's + // catch(...) boundary rather than a catch(std::exception&). + struct NonStdThrowingWriteSink : BodySink + { + bool write(const char*, std::size_t) override { throw 42; } + bool finish() override { return true; } + }; + + std::string trim_crlf(std::string s) + { + while (!s.empty() && (s.back() == '\n' || s.back() == '\r')) + s.pop_back(); + return s; + } +} // namespace + +TEST_CASE("request_body_file", "[http][body_file]") +{ + TempDir dir; + SimpleApp app; + app.max_body_size(1024 * 1024); + + CROW_ROUTE(app, "/memory") + .methods("POST"_method)([](const request& req) { + return std::string(FileBodySink::from(req) ? "file" : "memory") + ':' + req.body; + }); + + CROW_ROUTE(app, "/upload") + .methods("POST"_method) + .body_sink(FileBodySink::factory(dir.path.string()))([](const request& req) { + auto* file = FileBodySink::from(req); + if (!file) + return std::string("nobody:") + req.body; + return std::string("file:") + read_all(file->path()); + }); + + CROW_ROUTE(app, "/keep") + .methods("POST"_method) + .body_sink(FileBodySink::factory(dir.path.string()))([](const request& req) { + auto* file = FileBodySink::from(req); + file->keep(); + return file->path(); + }); + + const auto custom_dir = dir.path / "route-dir"; + std::filesystem::create_directories(custom_dir); + CROW_ROUTE(app, "/custom") + .methods("POST"_method) + .body_sink(FileBodySink::factory(custom_dir.string()))([](const request& req) { + return FileBodySink::from(req)->path(); + }); + + CROW_ROUTE(app, "/getfile") + .body_sink(FileBodySink::factory(dir.path.string()))([](const request& req) { + return FileBodySink::from(req) ? "file" : "nobody"; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + const auto port = app.port(); + + auto post = [&](const std::string& target, const std::string& payload, + const std::string& extra_headers = {}) { + TestClient client(port); + client.send( + "POST " + target + + " HTTP/1.1\r\n" + "Host: localhost\r\n" + + extra_headers + + "Content-Length: " + std::to_string(payload.size()) + + "\r\n" + "\r\n"); + if (!payload.empty()) + client.send(payload.data(), payload.size()); + return client.receive(); + }; + + const auto memory = post("/memory", "hello"); + CHECK(memory.find("HTTP/1.1 200") != std::string::npos); + CHECK(http_body(memory) == "memory:hello"); + + const std::string payload(64 * 1024, 'A'); + const auto uploaded = post("/upload", payload); + REQUIRE(uploaded.find("HTTP/1.1 200") != std::string::npos); + CHECK(http_body(uploaded) == "file:" + payload); + + const std::string binary(std::string("a") + '\0' + "b" + '\xff' + "c"); + const auto binary_response = post("/upload", binary); + REQUIRE(binary_response.find("HTTP/1.1 200") != std::string::npos); + CHECK(http_body(binary_response) == "file:" + binary); + + const auto empty = post("/upload", ""); + REQUIRE(empty.find("HTTP/1.1 200") != std::string::npos); + CHECK(http_body(empty) == "nobody:"); + + { + TestClient client(port); + client.send( + "GET /getfile HTTP/1.1\r\n" + "Host: localhost\r\n" + "\r\n"); + CHECK(http_body(client.receive()) == "nobody"); + CHECK(no_crow_body_files(dir.path)); + } + + { + TestClient client(port); + client.send( + "GET /getfile HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 0\r\n" + "\r\n"); + CHECK(http_body(client.receive()) == "nobody"); + CHECK(no_crow_body_files(dir.path)); + } + + { + // An empty *chunked* body still counts as an incoming body and opens + // the sink, unlike Content-Length: 0. + TestClient client(port); + client.send( + "POST /upload HTTP/1.1\r\n" + "Host: localhost\r\n" + "Transfer-Encoding: chunked\r\n" + "\r\n" + "0\r\n\r\n"); + CHECK(http_body(client.receive()) == "file:"); + } + + { + TestClient client(port); + client.send( + "POST /upload HTTP/1.1\r\n" + "Host: localhost\r\n" + "Transfer-Encoding: chunked\r\n" + "\r\n" + "5\r\nhello\r\n" + "6\r\n world\r\n" + "0\r\n\r\n"); + const auto response = client.receive(); + REQUIRE(response.find("HTTP/1.1 200") != std::string::npos); + CHECK(http_body(response) == "file:hello world"); + } + + const auto continued = post("/upload", "ping", "Expect: 100-continue\r\n"); + REQUIRE(continued.find("100 Continue") != std::string::npos); + REQUIRE(continued.find("HTTP/1.1 200") != std::string::npos); + CHECK(http_body(continued) == "file:ping"); + + { + const auto response = post("/keep", "abc"); + const auto kept = trim_crlf(http_body(response)); + REQUIRE(std::filesystem::exists(kept)); + CHECK(read_all(kept) == "abc"); +#ifndef _WIN32 + struct stat st {}; + REQUIRE(::stat(kept.c_str(), &st) == 0); + CHECK((st.st_mode & 0777) == 0600); +#endif + std::filesystem::remove(kept); + } + + { + const auto response = post("/custom", "Z"); + const auto custom_path = trim_crlf(http_body(response)); + CHECK(custom_path.find(custom_dir.string()) != std::string::npos); + } + + { + // Keep-alive: two uploads reusing the same socket both work, and the + // first upload's file is not disturbed by the second request. + TestClient client(port); + client.send( + "POST /upload HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 4\r\n" + "\r\n" + "one!"); + const auto first = client.receive(); + CHECK(http_body(first) == "file:one!"); + client.send( + "POST /upload HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 5\r\n" + "\r\n" + "two!!"); + const auto second = client.receive(); + CHECK(http_body(second) == "file:two!!"); + } + + const auto upload_path_resp = post("/keep", payload); + const auto first_path = trim_crlf(http_body(upload_path_resp)); + const auto second_keep = post("/keep", "x"); + const auto second_path = trim_crlf(http_body(second_keep)); + CHECK(first_path != second_path); + std::filesystem::remove(first_path); + std::filesystem::remove(second_path); + + app.stop(); +} + +TEST_CASE("request_body_file keep() survives a copied request", "[http][body_file]") +{ + // Regression test for the copy trap: the flag that used to live on + // `request` (persist_body_file_) was a plain bool copied by value, so + // calling the "keep" operation on a copy never reached the parser's own + // request and the file was deleted anyway. `body_sink` is a shared_ptr + // now, so every copy shares the same underlying FileBodySink. + TempDir dir; + SimpleApp app; + app.max_body_size(1024 * 1024); + + CROW_ROUTE(app, "/keep-via-copy") + .methods("POST"_method) + .body_sink(FileBodySink::factory(dir.path.string()))([](const request& req) { + crow::request copy = req; // a distinct object, sharing body_sink + FileBodySink::from(copy)->keep(); + return FileBodySink::from(req)->path(); + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + + TestClient client(app.port()); + client.send( + "POST /keep-via-copy HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 3\r\n" + "\r\n" + "abc"); + const auto path = trim_crlf(http_body(client.receive())); + REQUIRE(std::filesystem::exists(path)); + CHECK(read_all(path) == "abc"); + std::filesystem::remove(path); + + app.stop(); +} + +TEST_CASE("request_body_file deleted only after the full response is sent", "[http][body_file]") +{ + // Regression test: the file used to be unlinked as soon as the *first* + // chunk of a streamed response was written (parser_.clear() ran inside + // the per-chunk write helper), not after the whole response. Force the + // streaming path with a body well above the default 1MiB threshold, then + // pause the client mid-response (without finishing the read) and check + // the file is still on disk while more of it is still in flight. + TempDir dir; + SimpleApp app; + app.max_body_size(1024 * 1024); + app.stream_threshold(1024); + + const std::size_t response_body_size = 8 * 1024 * 1024; + CROW_ROUTE(app, "/upload") + .methods("POST"_method) + .body_sink(FileBodySink::factory(dir.path.string()))([response_body_size](const request&) { + return std::string(response_body_size, 'R'); + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + + TestClient client(app.port()); + client.send( + "POST /upload HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 3\r\n" + "\r\n" + "abc"); + + // Read the response headers one byte at a time so nothing beyond them is + // consumed, note the advertised Content-Length, then read a small prefix + // of the body and stop — leaving the server's write loop stalled with + // most of the body still unsent. + std::string headers; + while (headers.find("\r\n\r\n") == std::string::npos) + headers += client.read_some(1); + const auto length_pos = headers.find("Content-Length:"); + REQUIRE(length_pos != std::string::npos); + const auto content_length = static_cast(std::stoull(headers.substr(length_pos + 15))); + REQUIRE(content_length == response_body_size); + + const std::size_t prefix_size = 4096; + client.read_some(prefix_size); + std::this_thread::sleep_for(std::chrono::milliseconds(300)); + + CHECK_FALSE(directory_is_empty(dir.path)); + + // Drain exactly the rest of this one response (no more, so a bound read + // never blocks on the still-open keep-alive connection) so the server + // can finish and the test can clean up. + client.read_some(content_length - prefix_size); + + app.stop(); +} + +TEST_CASE("request_body_file_too_large", "[http][body_file]") +{ + TempDir dir; + std::atomic handler_ran{false}; + SimpleApp app; + app.max_body_size(8); + + CROW_ROUTE(app, "/upload") + .methods("POST"_method) + .body_sink(FileBodySink::factory(dir.path.string()))([&handler_ran](const request&) { + handler_ran = true; + return "ran"; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + + TestClient client(app.port()); + client.send( + "POST /upload HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 9\r\n" + "\r\n" + "123456789"); + const auto response = client.receive(); + CHECK(response.find("HTTP/1.1 413") != std::string::npos); + CHECK(response.find("Connection: close") != std::string::npos); + CHECK_FALSE(handler_ran.load()); + CHECK(directory_is_empty(dir.path)); + + app.stop(); +} + +TEST_CASE("request_body_file_too_large chunked, over limit on a sink route", "[http][body_file]") +{ + // The Content-Length fast path at headers-complete can't catch a chunked + // body (the size isn't known yet), so this exercises the over-limit + // check inside on_body, after the sink has already been opened. + TempDir dir; + std::atomic handler_ran{false}; + SimpleApp app; + app.max_body_size(8); + + CROW_ROUTE(app, "/upload") + .methods("POST"_method) + .body_sink(FileBodySink::factory(dir.path.string()))([&handler_ran](const request&) { + handler_ran = true; + return "ran"; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + + TestClient client(app.port()); + client.send( + "POST /upload HTTP/1.1\r\n" + "Host: localhost\r\n" + "Transfer-Encoding: chunked\r\n" + "\r\n" + "a\r\n0123456789\r\n" + "0\r\n\r\n"); + const auto response = client.receive(); + CHECK(response.find("HTTP/1.1 413") != std::string::npos); + CHECK(response.find("Connection: close") != std::string::npos); + CHECK_FALSE(handler_ran.load()); + + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(2); + bool empty = false; + while (std::chrono::steady_clock::now() < deadline) + { + empty = directory_is_empty(dir.path); + if (empty) + break; + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + CHECK(empty); + + app.stop(); +} + +TEST_CASE("request_body_file_disconnect_cleans_up", "[http][body_file]") +{ + TempDir dir; + std::atomic handler_ran{false}; + SimpleApp app; + + CROW_ROUTE(app, "/upload") + .methods("POST"_method) + .body_sink(FileBodySink::factory(dir.path.string()))([&handler_ran](const request&) { + handler_ran = true; + return "ran"; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + + { + TestClient client(app.port()); + client.send( + "POST /upload HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 64\r\n" + "\r\n" + "partial"); + } + + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(2); + bool empty = false; + while (std::chrono::steady_clock::now() < deadline) + { + empty = directory_is_empty(dir.path); + if (empty) + break; + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + CHECK(empty); + CHECK_FALSE(handler_ran.load()); + + app.stop(); +} + +TEST_CASE("request_body_sink", "[http][body_file]") +{ + auto buf = std::make_shared(); + std::atomic body_size{999}; + std::atomic body_capacity{999}; + std::atomic had_file{true}; + SimpleApp app; + app.max_body_size(1024 * 1024); + + CROW_ROUTE(app, "/sink") + .methods("POST"_method) + .body_sink([buf](const request&) { + auto sink = std::make_unique(); + sink->buf = buf; + return sink; + })([buf, &body_size, &body_capacity, &had_file](const request& req) { + body_size = req.body.size(); + body_capacity = req.body.capacity(); + had_file = static_cast(FileBodySink::from(req)); + return req.body.empty() ? *buf : req.body; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + + const std::string payload(32 * 1024, 'S'); + TestClient client(app.port()); + client.send( + "POST /sink HTTP/1.1\r\nHost: localhost\r\nContent-Length: " + + std::to_string(payload.size()) + "\r\n\r\n" + payload); + CHECK(http_body(client.receive()) == payload); + CHECK(body_size.load() == 0); + CHECK(body_capacity.load() < payload.size()); + CHECK_FALSE(had_file.load()); + + app.stop(); +} + +TEST_CASE("request_body_file does not reserve req.body", "[http][body_file]") +{ + TempDir dir; + std::atomic body_size{999}; + std::atomic body_capacity{999}; + SimpleApp app; + app.max_body_size(1024 * 1024); + + CROW_ROUTE(app, "/upload") + .methods("POST"_method) + .body_sink(FileBodySink::factory(dir.path.string()))([&](const request& req) { + body_size = req.body.size(); + body_capacity = req.body.capacity(); + return "ok"; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + + const std::string payload(64 * 1024, 'A'); + TestClient client(app.port()); + client.send( + "POST /upload HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: " + + std::to_string(payload.size()) + + "\r\n" + "\r\n"); + client.send(payload.data(), payload.size()); + const auto response = client.receive(); + REQUIRE(response.find("HTTP/1.1 200") != std::string::npos); + CHECK(body_size.load() == 0); + CHECK(body_capacity.load() < payload.size()); + + app.stop(); +} + +TEST_CASE("request_body_sink write failure is 500", "[http][body_file]") +{ + std::atomic handler_ran{false}; + SimpleApp app; + app.max_body_size(1024 * 1024); + + CROW_ROUTE(app, "/sink") + .methods("POST"_method) + .body_sink([](const request&) { + return std::make_unique(); + })([&handler_ran](const request&) { + handler_ran = true; + return "ran"; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + + TestClient client(app.port()); + client.send( + "POST /sink HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 4\r\n" + "\r\n" + "fail"); + const auto response = client.receive(); + CHECK(response.find("HTTP/1.1 500") != std::string::npos); + CHECK(response.find("Connection: close") != std::string::npos); + CHECK_FALSE(handler_ran.load()); + + app.stop(); +} + +TEST_CASE("request_body_sink write failure lets a large in-flight upload finish writing before the 500 is read", "[http][body_file]") +{ + // Regression test: a client that writes its whole over-limit body before + // reading must not see its write fail (e.g. with a broken pipe). The + // server must linger and keep draining the socket instead of closing on + // unread bytes, which can RST a peer that is still mid-upload and lose + // the response it already sent. A small payload (4 bytes, as in the test + // above) cannot reproduce this: it needs to be big enough that the + // client's blocking send() has to wait on the server to keep reading, + // rather than completing into socket buffers before the server has even + // reacted. + std::atomic handler_ran{false}; + SimpleApp app; + app.max_body_size(64ull * 1024 * 1024); + + CROW_ROUTE(app, "/sink") + .methods("POST"_method) + .body_sink([](const request&) { + return std::make_unique(); + })([&handler_ran](const request&) { + handler_ran = true; + return "ran"; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + + const std::string payload(5'000'000, 'x'); + TestClient client(app.port()); + const std::string http_request = + "POST /sink HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: " + + std::to_string(payload.size()) + + "\r\n" + "\r\n" + + payload; + + REQUIRE_NOTHROW(client.send(http_request)); + const auto response = client.receive(); + CHECK(response.find("HTTP/1.1 500") != std::string::npos); + CHECK_FALSE(handler_ran.load()); + + app.stop(); +} + +TEST_CASE("request_body_sink finish failure is 500", "[http][body_file]") +{ + std::atomic handler_ran{false}; + SimpleApp app; + app.max_body_size(1024 * 1024); + + CROW_ROUTE(app, "/sink") + .methods("POST"_method) + .body_sink([](const request&) { + return std::make_unique(); + })([&handler_ran](const request&) { + handler_ran = true; + return "ran"; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + + TestClient client(app.port()); + client.send( + "POST /sink HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 4\r\n" + "\r\n" + "fail"); + const auto response = client.receive(); + CHECK(response.find("HTTP/1.1 500") != std::string::npos); + CHECK(response.find("Connection: close") != std::string::npos); + CHECK_FALSE(handler_ran.load()); + + app.stop(); +} + +TEST_CASE("request_body_sink finish failure sends exactly one response", "[http][body_file]") +{ + // Regression test: on_message_complete() must not fall through to + // process_message() (a second handle() call) after reject_body() already + // wrote the 500 for a finish() failure - that would put two status lines + // on one connection. + SimpleApp app; + app.max_body_size(1024 * 1024); + + CROW_ROUTE(app, "/sink") + .methods("POST"_method) + .body_sink([](const request&) { + return std::make_unique(); + })([](const request&) { + return "ran"; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + + TestClient client(app.port()); + client.send( + "POST /sink HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 4\r\n" + "\r\n" + "fail"); + const auto response = client.receive(); + CHECK(response.find("HTTP/1.1 500") != std::string::npos); + + const auto leftover = client.read_leftover(std::chrono::milliseconds(200)); + CHECK(leftover.empty()); + + app.stop(); +} + +TEST_CASE("request_body_sink a throwing write() or finish() is 500", "[http][body_file]") +{ + // Regression test: write()/finish() used to be called bare; an uncaught + // exception unwound into the worker loop's catch(std::exception&), + // logged "Worker Crash" and left the client with no response at all + // (and a non-std::exception ended the worker thread silently). + std::atomic handler_ran{false}; + SimpleApp app; + app.max_body_size(1024 * 1024); + + CROW_ROUTE(app, "/throw-write") + .methods("POST"_method) + .body_sink([](const request&) { + return std::make_unique(); + })([&handler_ran](const request&) { + handler_ran = true; + return "ran"; + }); + + CROW_ROUTE(app, "/throw-finish") + .methods("POST"_method) + .body_sink([](const request&) { + return std::make_unique(); + })([&handler_ran](const request&) { + handler_ran = true; + return "ran"; + }); + + CROW_ROUTE(app, "/throw-write-nonstd") + .methods("POST"_method) + .body_sink([](const request&) { + return std::make_unique(); + })([&handler_ran](const request&) { + handler_ran = true; + return "ran"; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + const auto port = app.port(); + + auto post_fail = [&](const std::string& target) { + TestClient client(port); + client.send( + "POST " + target + + " HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 4\r\n" + "\r\n" + "fail"); + return client.receive(); + }; + + for (const std::string target : {"/throw-write", "/throw-finish", "/throw-write-nonstd"}) + { + handler_ran = false; + const auto response = post_fail(target); + CHECK(response.find("HTTP/1.1 500") != std::string::npos); + CHECK(response.find("Connection: close") != std::string::npos); + CHECK_FALSE(handler_ran.load()); + } + + // The server is still responsive: a throw doesn't take down the worker. + TestClient probe(port); + probe.send( + "POST /throw-write HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 1\r\n" + "\r\n" + "x"); + CHECK(probe.receive().find("HTTP/1.1 500") != std::string::npos); + + app.stop(); +} + +TEST_CASE("request_body_sink calling body_sink() again replaces the factory", "[http][body_file]") +{ + TempDir dir; + auto buf = std::make_shared(); + SimpleApp app; + app.max_body_size(1024 * 1024); + + CROW_ROUTE(app, "/last-wins") + .methods("POST"_method) + .body_sink(FileBodySink::factory(dir.path.string())) + .body_sink([buf](const request&) { + auto sink = std::make_unique(); + sink->buf = buf; + return sink; + })([buf](const request& req) { + return FileBodySink::from(req) ? std::string("file") : *buf; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + + TestClient client(app.port()); + client.send( + "POST /last-wins HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 3\r\n" + "\r\n" + "abc"); + CHECK(http_body(client.receive()) == "abc"); + CHECK(directory_is_empty(dir.path)); + + app.stop(); +} + +TEST_CASE("request_body_sink a factory returning nullptr keeps the body in req.body", "[http][body_file]") +{ + // A user factory declining (nullptr) is not an error: unlike the file + // sink's own open failure, it means "use req.body as usual" for this + // request. + std::atomic handler_ran{false}; + SimpleApp app; + app.max_body_size(1024 * 1024); + + CROW_ROUTE(app, "/maybe") + .methods("POST"_method) + .body_sink([](const request&) -> std::unique_ptr { + return nullptr; + })([&handler_ran](const request& req) { + handler_ran = true; + return std::string("memory:") + req.body; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + + TestClient client(app.port()); + client.send( + "POST /maybe HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 5\r\n" + "\r\n" + "hello"); + const auto response = client.receive(); + CHECK(response.find("HTTP/1.1 200") != std::string::npos); + CHECK(http_body(response) == "memory:hello"); + CHECK(handler_ran.load()); + + app.stop(); +} + +TEST_CASE("request_body_sink a throwing factory is 500", "[http][body_file]") +{ + std::atomic handler_ran{false}; + SimpleApp app; + app.max_body_size(1024 * 1024); + + CROW_ROUTE(app, "/throw") + .methods("POST"_method) + .body_sink([](const request&) -> std::unique_ptr { + throw std::runtime_error("sink open"); + })([&handler_ran](const request&) { + handler_ran = true; + return "ran"; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + + TestClient client(app.port()); + client.send( + "POST /throw HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 4\r\n" + "\r\n" + "fail"); + const auto response = client.receive(); + CHECK(response.find("HTTP/1.1 500") != std::string::npos); + CHECK(response.find("Connection: close") != std::string::npos); + CHECK_FALSE(handler_ran.load()); + + app.stop(); +} + +TEST_CASE("FileBodySink::factory rejects a directory that does not exist", "[http][body_file]") +{ + TempDir dir; + const auto missing = dir.path / "does-not-exist"; + CHECK_THROWS_AS(FileBodySink::factory(missing.string()), std::filesystem::filesystem_error); + + const auto not_a_dir = dir.path / "not-a-dir"; + { + std::ofstream out(not_a_dir); + out << "x"; + } + CHECK_THROWS_AS(FileBodySink::factory(not_a_dir.string()), std::filesystem::filesystem_error); +} + +TEST_CASE("request_body_file per-request open failure is 500 and leaves no file", "[http][body_file]") +{ + // The directory exists (and is required to) when the route is set up; + // simulate it disappearing before a request actually opens a file. + TempDir dir; + const auto vanishing = dir.path / "vanishing"; + std::filesystem::create_directories(vanishing); + auto factory = FileBodySink::factory(vanishing.string()); + std::filesystem::remove(vanishing); + + std::atomic handler_ran{false}; + SimpleApp app; + app.max_body_size(1024 * 1024); + + CROW_ROUTE(app, "/upload") + .methods("POST"_method) + .body_sink(std::move(factory))([&handler_ran](const request&) { + handler_ran = true; + return "ran"; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + + TestClient client(app.port()); + client.send( + "POST /upload HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 4\r\n" + "\r\n" + "fail"); + const auto response = client.receive(); + CHECK(response.find("HTTP/1.1 500") != std::string::npos); + CHECK(response.find("Connection: close") != std::string::npos); + CHECK_FALSE(handler_ran.load()); + CHECK_FALSE(std::filesystem::exists(vanishing)); + + app.stop(); +} + +TEST_CASE("request_body_file concurrent uploads get distinct paths", "[http][body_file]") +{ + TempDir dir; + SimpleApp app; + app.max_body_size(1024 * 1024); + + CROW_ROUTE(app, "/keep") + .methods("POST"_method) + .body_sink(FileBodySink::factory(dir.path.string()))([](const request& req) { + auto* file = FileBodySink::from(req); + file->keep(); + return file->path(); + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + const auto port = app.port(); + + TestClient a(port); + TestClient b(port); + a.send( + "POST /keep HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 8\r\n" + "\r\n" + "AAAA"); + b.send( + "POST /keep HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 8\r\n" + "\r\n" + "BBBB"); + + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(2); + std::size_t nfiles = 0; + while (std::chrono::steady_clock::now() < deadline) + { + nfiles = 0; + std::error_code ec; + for (const auto& entry : std::filesystem::directory_iterator(dir.path, ec)) + { + if (!ec && entry.is_regular_file()) + ++nfiles; + } + if (nfiles >= 2) + break; + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + REQUIRE(nfiles >= 2); + + a.send("aaaa"); + b.send("bbbb"); + const auto path_a = trim_crlf(http_body(a.receive())); + const auto path_b = trim_crlf(http_body(b.receive())); + CHECK(path_a != path_b); + CHECK(read_all(path_a) == "AAAAaaaa"); + CHECK(read_all(path_b) == "BBBBbbbb"); + std::filesystem::remove(path_a); + std::filesystem::remove(path_b); + + app.stop(); +} diff --git a/tests/fuzz/http_fuzzer.cpp b/tests/fuzz/http_fuzzer.cpp index c77956ca26..6690d6c86b 100644 --- a/tests/fuzz/http_fuzzer.cpp +++ b/tests/fuzz/http_fuzzer.cpp @@ -4,8 +4,10 @@ struct DummyHandler { void handle_url() {} - void handle_header() {} + int handle_header() { return 0; } void handle() {} + void reject_body(int) {} + bool parser_should_abort() const { return false; } size_t stream_threshold() { return 1024*1024; } }; diff --git a/tests/http_test_utils.h b/tests/http_test_utils.h new file mode 100644 index 0000000000..5503b4e15b --- /dev/null +++ b/tests/http_test_utils.h @@ -0,0 +1,166 @@ +#pragma once + +// Shared helpers for tests that drive a running Crow server over a raw +// socket (max_body_size_tests.cpp, body_file_tests.cpp). Free functions are +// `inline` because this header is included by more than one translation +// unit linked into the same `unittest` binary. + +#include +#include +#include +#include +#include + +#include "catch2/catch_all.hpp" + +namespace crow_test_utils +{ + // True once `data` holds a full response (status line through body, per + // Content-Length; a response with none is complete at the header + // terminator). Skips past any 1xx interim responses (e.g. 100 Continue). + inline bool response_complete(const std::string& data) + { + auto search_from = std::size_t{0}; + while (true) + { + const auto status_line = data.find("HTTP/1.1 ", search_from); + if (status_line == std::string::npos) + return false; + const auto code = std::atoi(data.c_str() + status_line + 9); + const auto header_end = data.find("\r\n\r\n", status_line); + if (header_end == std::string::npos) + return false; + if (code >= 100 && code < 200) + { + search_from = header_end + 4; + continue; + } + const auto length_pos = data.find("Content-Length:", status_line); + if (length_pos == std::string::npos || length_pos > header_end) + return true; + const auto length = static_cast(std::stoull(data.substr(length_pos + 15))); + return data.size() >= header_end + 4 + length; + } + } + + // The status code of the last non-interim response in `response`. + inline int status_of(const std::string& response) + { + auto search_from = std::size_t{0}; + while (true) + { + const auto status_line = response.find("HTTP/1.1 ", search_from); + if (status_line == std::string::npos) + return 0; + const auto code = std::atoi(response.c_str() + status_line + 9); + if (code >= 100 && code < 200) + { + search_from = status_line + 9; + continue; + } + return code; + } + } + + // The body of the last non-interim response in `response`. + inline std::string http_body(const std::string& response) + { + auto search_from = std::size_t{0}; + while (true) + { + const auto status_line = response.find("HTTP/1.1 ", search_from); + if (status_line == std::string::npos) + return {}; + const auto code = std::atoi(response.c_str() + status_line + 9); + const auto header_end = response.find("\r\n\r\n", status_line); + if (header_end == std::string::npos) + return {}; + if (code >= 100 && code < 200) + { + search_from = header_end + 4; + continue; + } + return response.substr(header_end + 4); + } + } + + // A bare-socket HTTP client, for tests that need to control framing + // (partial sends, reading mid-response, checking what's left on the wire) + // more precisely than a real HTTP client would allow. + // + // Relies on `asio`/`asio_error_code` and `LOCALHOST_ADDRESS` already + // being visible at the point this header is included (each including + // .cpp sets those up first, matching the CROW_USE_BOOST/standalone asio + // switch already in effect for the rest of the file). + class TestClient + { + public: + TestClient(uint16_t port): + socket_(io_context_) + { + socket_.connect(asio::ip::tcp::endpoint(asio::ip::make_address(LOCALHOST_ADDRESS), port)); + } + + void send(const std::string& data) + { + asio::write(socket_, asio::buffer(data)); + } + + void send(const char* data, std::size_t size) + { + asio::write(socket_, asio::buffer(data, size)); + } + + // Reads exactly `size` more bytes into an internal buffer without + // waiting for the response to complete. Used to pause a client + // mid-response and observe server-side state while it is still + // streaming. + std::string read_some(std::size_t size) + { + std::string out(size, '\0'); + asio::read(socket_, asio::buffer(out)); + return out; + } + + // Collects whatever bytes show up within `budget`, to check for a + // second response wrongly sent after a first one is already complete. + std::string read_leftover(std::chrono::milliseconds budget) + { + std::string leftover; + std::array buffer{}; + socket_.non_blocking(true); + const auto deadline = std::chrono::steady_clock::now() + budget; + while (std::chrono::steady_clock::now() < deadline) + { + asio_error_code ec; + const auto n = socket_.read_some(asio::buffer(buffer), ec); + if (!ec && n > 0) + leftover.append(buffer.data(), n); + else + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + } + return leftover; + } + + std::string receive() + { + std::string response; + std::array buffer{}; + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(5); + while (!response_complete(response)) + { + REQUIRE(std::chrono::steady_clock::now() < deadline); + asio_error_code ec; + const auto n = socket_.read_some(asio::buffer(buffer), ec); + if (ec) + break; + response.append(buffer.data(), n); + } + return response; + } + + private: + asio::io_context io_context_{}; + asio::ip::tcp::socket socket_; + }; +} // namespace crow_test_utils diff --git a/tests/max_body_size_tests.cpp b/tests/max_body_size_tests.cpp new file mode 100644 index 0000000000..4f217c7093 --- /dev/null +++ b/tests/max_body_size_tests.cpp @@ -0,0 +1,456 @@ +#define CROW_ENABLE_DEBUG +#define CROW_LOG_LEVEL 0 + +#include +#include +#include +#include +#include +#include + +#include "catch2/catch_all.hpp" +#include "crow.h" +#include "crow/middlewares/cors.h" + +using namespace crow; + +#ifdef CROW_USE_BOOST +namespace asio = boost::asio; +using asio_error_code = boost::system::error_code; +#else +using asio_error_code = asio::error_code; +#endif + +#define LOCALHOST_ADDRESS "127.0.0.1" + +#include "http_test_utils.h" +using namespace crow_test_utils; + +TEST_CASE("max_body_size advertised length", "[http][max_body_size]") +{ + std::atomic handler_ran{false}; + SimpleApp app; + app.max_body_size(8); + + CROW_ROUTE(app, "/upload") + .methods("POST"_method)([&handler_ran](const request& req) { + handler_ran = true; + return req.body; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + const auto port = app.port(); + + auto post = [&](const std::string& target, const std::string& payload, + const std::string& extra = {}) { + TestClient client(port); + client.send( + "POST " + target + + " HTTP/1.1\r\n" + "Host: localhost\r\n" + + extra + + "Content-Length: " + std::to_string(payload.size()) + + "\r\n" + "\r\n" + + payload); + return client.receive(); + }; + + SECTION("body of exactly max is accepted") + { + const auto resp = post("/upload", "12345678"); + CHECK(status_of(resp) == 200); + CHECK(resp.find("12345678") != std::string::npos); + CHECK(handler_ran); + } + + SECTION("Content-Length above the cap is 413 at headers and does not run the handler") + { + handler_ran = false; + const auto resp = post("/upload", "123456789"); + CHECK(status_of(resp) == 413); + CHECK(resp.find("Connection: close") != std::string::npos); + CHECK_FALSE(handler_ran); + } + + SECTION("Expect 100-continue is not sent for an over-limit body") + { + handler_ran = false; + TestClient client(port); + client.send( + "POST /upload HTTP/1.1\r\n" + "Host: localhost\r\n" + "Expect: 100-continue\r\n" + "Content-Length: 5000000\r\n" + "\r\n"); + const auto resp = client.receive(); + CHECK(resp.find("100 Continue") == std::string::npos); + CHECK(status_of(resp) == 413); + CHECK_FALSE(handler_ran); + } + + SECTION("unmatched URL still applies the app limit") + { + handler_ran = false; + const auto resp = post("/missing", std::string(100, 'x')); + CHECK(status_of(resp) == 413); + CHECK_FALSE(handler_ran); + } + + SECTION("wrong method still applies the app limit") + { + handler_ran = false; + TestClient client(port); + client.send( + "PUT /upload HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 100\r\n" + "\r\n" + + std::string(100, 'x')); + const auto resp = client.receive(); + CHECK(status_of(resp) == 413); + CHECK_FALSE(handler_ran); + } + + app.stop(); +} + +TEST_CASE("max_body_size slash redirect", "[http][max_body_size]") +{ + std::atomic handler_ran{false}; + SimpleApp app; + app.max_body_size(8); + + CROW_ROUTE(app, "/probe/") + .methods("POST"_method)([&handler_ran] { + handler_ran = true; + return "ok"; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + + TestClient client(app.port()); + client.send( + "POST /probe HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 100\r\n" + "\r\n" + + std::string(100, 'x')); + const auto resp = client.receive(); + CHECK(status_of(resp) == 413); + CHECK_FALSE(handler_ran); + + app.stop(); +} + +TEST_CASE("max_body_size chunked accumulate", "[http][max_body_size]") +{ + std::atomic handler_ran{false}; + SimpleApp app; + app.max_body_size(8); + + CROW_ROUTE(app, "/upload") + .methods("POST"_method)([&handler_ran](const request&) { + handler_ran = true; + return "ok"; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + + TestClient client(app.port()); + client.send( + "POST /upload HTTP/1.1\r\n" + "Host: localhost\r\n" + "Transfer-Encoding: chunked\r\n" + "\r\n" + "5\r\nhello\r\n" + "5\r\nworld\r\n" + "0\r\n" + "\r\n"); + const auto resp = client.receive(); + CHECK(status_of(resp) == 413); + CHECK_FALSE(handler_ran); + + app.stop(); +} + +TEST_CASE("max_body_size per-route override", "[http][max_body_size]") +{ + SimpleApp app; + app.max_body_size(8); + + CROW_ROUTE(app, "/small") + .methods("POST"_method)([](const request& req) { + return req.body; + }); + CROW_ROUTE(app, "/large") + .methods("POST"_method) + .max_body_size(32)([](const request& req) { + return req.body; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + const auto port = app.port(); + + const std::string payload(16, 'A'); + { + TestClient client(port); + client.send( + "POST /small HTTP/1.1\r\nHost: localhost\r\nContent-Length: 16\r\n\r\n" + payload); + CHECK(status_of(client.receive()) == 413); + } + { + TestClient client(port); + client.send( + "POST /large HTTP/1.1\r\nHost: localhost\r\nContent-Length: 16\r\n\r\n" + payload); + CHECK(status_of(client.receive()) == 200); + } + + app.stop(); +} + +TEST_CASE("max_body_size default unlimited", "[http][max_body_size]") +{ + SimpleApp app; + CROW_ROUTE(app, "/upload") + .methods("POST"_method)([](const request& req) { + return std::to_string(req.body.size()); + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + + const std::string payload(64 * 1024, 'B'); + TestClient client(app.port()); + client.send( + "POST /upload HTTP/1.1\r\nHost: localhost\r\nContent-Length: " + + std::to_string(payload.size()) + "\r\n\r\n" + payload); + const auto resp = client.receive(); + CHECK(status_of(resp) == 200); + CHECK(resp.find("65536") != std::string::npos); + + app.stop(); +} + +TEST_CASE("max_body_size 413 runs after-handlers", "[http][max_body_size]") +{ + App app; + app.max_body_size(8); + + CROW_ROUTE(app, "/upload") + .methods("POST"_method)([] { + return "ok"; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + + TestClient client(app.port()); + client.send( + "POST /upload HTTP/1.1\r\n" + "Host: localhost\r\n" + "Origin: https://example.test\r\n" + "Content-Length: 100\r\n" + "\r\n" + + std::string(100, 'x')); + const auto resp = client.receive(); + CHECK(status_of(resp) == 413); + CHECK(resp.find("Access-Control-Allow-Origin: *") != std::string::npos); + + app.stop(); +} + +TEST_CASE("max_body_size unlimited does not allocate advertised Content-Length", "[http][max_body_size]") +{ + SimpleApp app; + CROW_ROUTE(app, "/upload") + .methods("POST"_method)([](const request& req) { + return std::to_string(req.body.size()); + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + const auto port = app.port(); + + // Keep this socket open so handle_header actually sees the huge advertised + // length. Closing immediately can hide a reserve() that only runs after + // the server reads the headers. + TestClient attacker(port); + attacker.send( + "POST /upload HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 1125899906842624\r\n" + "\r\n"); + std::this_thread::sleep_for(std::chrono::milliseconds(200)); + + TestClient client(port); + client.send( + "POST /upload HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 4\r\n" + "\r\n" + "ping"); + const auto resp = client.receive(); + CHECK(status_of(resp) == 200); + CHECK(resp.find("4") != std::string::npos); + + app.stop(); +} + +TEST_CASE("max_body_size 413 skips before-handlers", "[http][max_body_size]") +{ + struct ProbeMiddleware + { + std::atomic before{false}; + std::atomic after{false}; + + struct context + {}; + + void before_handle(request& /*req*/, response& /*res*/, context& /*ctx*/) + { + before = true; + } + + void after_handle(request& /*req*/, response& /*res*/, context& /*ctx*/) + { + after = true; + } + }; + + App app; + app.max_body_size(8); + + CROW_ROUTE(app, "/upload") + .methods("POST"_method)([] { + return "ok"; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + + TestClient client(app.port()); + client.send( + "POST /upload HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 100\r\n" + "\r\n" + + std::string(100, 'x')); + const auto resp = client.receive(); + CHECK(status_of(resp) == 413); + + auto& probe = app.get_middleware(); + CHECK_FALSE(probe.before.load()); + CHECK(probe.after.load()); + + probe.before = false; + probe.after = false; + TestClient ok_client(app.port()); + ok_client.send( + "POST /upload HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 4\r\n" + "\r\n" + "abcd"); + CHECK(status_of(ok_client.receive()) == 200); + CHECK(probe.before.load()); + CHECK(probe.after.load()); + + app.stop(); +} + +TEST_CASE("max_body_size applies to a matched-route HEAD/OPTIONS request", "[http][max_body_size]") +{ + // A HEAD/OPTIONS request that matches a route reaches the max_body_size + // check in handle_header(). A HEAD 413 must still carry no body + // (skip_body set explicitly in the body_error_status_ branch, not left + // to accident), while OPTIONS gets a normal 413 body. + SimpleApp app; + app.max_body_size(8); + + CROW_ROUTE(app, "/upload") + .methods("GET"_method)([] { + return "ok"; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + const auto port = app.port(); + + { + TestClient client(port); + client.send( + "HEAD /upload HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 100\r\n" + "\r\n"); + const auto resp = client.receive(); + CHECK(status_of(resp) == 413); + CHECK(resp.find("Connection: close") != std::string::npos); + CHECK(resp.find("Content-Length: 0") != std::string::npos); + const auto header_end = resp.find("\r\n\r\n"); + REQUIRE(header_end != std::string::npos); + CHECK(resp.size() == header_end + 4); // no body after the headers + } + { + TestClient client(port); + client.send( + "OPTIONS /upload HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 100\r\n" + "\r\n"); + const auto resp = client.receive(); + CHECK(status_of(resp) == 413); + CHECK(resp.find("Connection: close") != std::string::npos); + } + + app.stop(); +} + +TEST_CASE("max_body_size 413 lingers instead of closing even with a small stream_threshold", "[http][max_body_size]") +{ + // Regression test: complete_request() has two response-writing paths, + // chosen by stream_threshold(), and both used to shut the socket down + // outright when close_connection_ was set - including for a 413/500 + // body-error response, racing ahead of do_read()'s linger_close() drain. + // A default-sized threshold (1 MiB) never routes a small error body + // through the streamed path, so the race was latent unless an app set a + // small stream_threshold(); this pins it down explicitly. + std::atomic handler_ran{false}; + SimpleApp app; + app.max_body_size(8); + app.stream_threshold(1); // force even the tiny 413 body through the streamed path + + CROW_ROUTE(app, "/upload") + .methods("POST"_method)([&handler_ran](const request& req) { + handler_ran = true; + return req.body; + }); + + auto server = app.bindaddr(LOCALHOST_ADDRESS).port(0).run_async(); + app.wait_for_server_start(); + const auto port = app.port(); + + TestClient client(port); + // Headers only, declaring a body well over the cap that the client sends next. + client.send( + "POST /upload HTTP/1.1\r\n" + "Host: localhost\r\n" + "Content-Length: 5000000\r\n" + "\r\n"); + const auto resp = client.receive(); + CHECK(status_of(resp) == 413); + CHECK_FALSE(handler_ran); + + // The client can still write its whole declared body without the write + // failing (broken pipe / connection reset), because the server drains + // instead of closing on unread bytes. + const std::string payload(5'000'000, 'x'); + REQUIRE_NOTHROW(client.send(payload)); + + app.stop(); +} diff --git a/tests/unittest.cpp b/tests/unittest.cpp index 1e60242bd2..8ecdf3f00d 100644 --- a/tests/unittest.cpp +++ b/tests/unittest.cpp @@ -3,6 +3,7 @@ #include #include +#include #include #include #include @@ -1892,6 +1893,17 @@ TEST_CASE("multipart_view") TEST_CASE("send_file") { + // CROW_STATIC_FILE's internal path is deliberately relative (its sanitizer + // strips a leading '/' to reject Unix absolute paths), so "tests/img/..." + // can't be replaced with an absolute path. Instead, chdir into the + // CMake-provided repo root for the scope of this test, so the relative + // paths below resolve regardless of where `unittest` is invoked from. + struct ScopedCwd + { + std::filesystem::path previous = std::filesystem::current_path(); + ScopedCwd(const std::filesystem::path& dir) { std::filesystem::current_path(dir); } + ~ScopedCwd() { std::filesystem::current_path(previous); } + } scoped_cwd(CROW_TEST_REPO_DIR); struct stat statbuf_cat; stat("tests/img/cat.jpg", &statbuf_cat);