Currently the "add new files during scanning" call stack is smaller than
the "remove deleted files during scanning" call stack. This can lead to
the scanner adding folders in the folder tree that are to deep to be
removed.
This changes the `removeChildren` logic to be non recursive so there is
no limit to the depth of the folder tree during removal
Signed-off-by: Robin Appelman <robin@icewind.nl>
Fixes#16876
Before we'd just fetch everything from all storages we'd have access to.
Then we'd sort. And filter in php. Now this of course is tricky if a
user shared just a file with you and then has a ton of activity.
Now we try to contruct the prefix path. So that the filtering can happen
right away in the databae.
Now this will make the DB more busy. But it should help overall as in
most cases less queries are needed then etc.
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
* introduces a new IRootMountProvider to register mount points inside the root storage
* adds a AppdataPreviewObjectStoreStorage to handle the split between preview folders and bucket number
Ref #22033
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
having the "cache rename" after the "storage move" caused the target
to get the fileid from the source file, without taking care that the object
is stored under the original file id.
By doing the "cache rename" first, we trigger the "update existing file"
logic while moving the file to the object store and the object gets stored for the
correct file id
Signed-off-by: Robin Appelman <robin@icewind.nl>
"Exception: substr() expects parameter 3 to be int, bool given" can occur on Line 378 $mimePart = substr($icon, 0, strpos($icon, '-'));
This happens, when '-' is not found and strpos returns false instead of an int.
When this occurs, e.g., Activity hangs.
Signed-off-by: lui87kw <lukas.ifflaender@uni-wuerzburg.de>
the php ftp streamwrapper doesn't handle hashes correctly and will break when it tries to enter a path containing a hash.
By filtering out paths containing a hash we can at least stop the external storage from breaking completely
Signed-off-by: Robin Appelman <robin@icewind.nl>
while some code paths do wrap the "raw" locking exception into one with a proper path, not all of them do
by adding the proper path to the original exception we ensure that we always have the usefull information in out logs
Signed-off-by: Robin Appelman <robin@icewind.nl>
The S3 client enables this by default and then tries to read
`.aws/config`. This causes `open_basedir` restriction related error for
some setups. So this patch disables the CSM because it's most likely
unused anyway.
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
while this scan *should* never be triggered, it's good to have some failsafe to ensure
that the users home contents don't end up getting scanned in the root storage
Signed-off-by: Robin Appelman <robin@icewind.nl>
Some S3 providers need a custom upload part size (500 MB static value in Nextcloud).
Here is a commit to change this value via S3 configuration, instead of using S3_UPLOAD_PART_SIZE constant.
A new parameter is added for an S3 connection : uploadPartSize
Signed-off-by: Florent <florent@coppint.com>
Right now if you want to get events via the Node API you have to have a
real instance of the Root. Which in turns sets up the whole FS.
We should make sure this is done lazy. Else enabling the preview
generator for example makes you setup the whole FS on each and every
authenticated call.
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
Else if a lot of writes happen. It might happen that an old stat result
is used. Resulting in a wrong file size for the file. For example the
text app when a lot of people edit at the same time.
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
Until now, you wouldn't be able to create
objects larger that 5GB.
It's somewhat related with pull #18883
Signed-off-by: Adrian Brzezinski <adrian.brzezinski@eo.pl>
Improves efficiency when downloading files from Swift storage.
Before, files were downloaded and then pushed back to user.
That behaevior causes all kinds of performance problems.
Now, files are streamed directly to user.
Signed-off-by: Adrian Brzezinski <adrian.brzezinski@eo.pl>
Currently you need to use `opendir` and then call `getMetadata` for
every file, which adds overhead because most storage backends already
get the metadata when doing the `opendir`.
While storagebackends can (and do) use caching to relief this problem,
this adds cache invalidation dificulties and only a limited number of
items are generally cached (to prevent memory usage exploding when
scanning large storages)
With this new methods storage backends can use the child metadata they
got from listing the folder to return metadata without having to keep
seperate caches.
Signed-off-by: Robin Appelman <robin@icewind.nl>
hashes are set in "X-Hash-MD5", "X-Hash-SHA1" and "X-Hash-SHA256" headers.
these headers are set for file uploads and the MOVE request at the end of a multipart upload.
Signed-off-by: Robin Appelman <robin@icewind.nl>
To continue this formatting madness, here's a tiny patch that adds
unified formatting for control structures like if and loops as well as
classes, their methods and anonymous functions. This basically forces
the constructs to start on the same line. This is not exactly what PSR2
wants, but I think we can have a few exceptions with "our" style. The
starting of braces on the same line is pracrically standard for our
code.
This also removes and empty lines from method/function bodies at the
beginning and end.
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
instead of first creating an empty file and then writing the content.
This solves the overhead of creating an empty file with the common pattern:
```php
$file = $simpleFilder->newFile('foo.txt');
$file->putContent('bar.txt');
```
roughly halving the number of storage and database operations that need to be done when creating a `SimpleFile`.
This is not automatically done with `File` because that has a more complex api which I'm more hesitant to touch.
Instead the `Folder::newFile` api has been extended to accept the content for the new file.
In my local testing, the overhead of first creating an empty file took about 20% of the time for preview generation
Signed-off-by: Robin Appelman <robin@icewind.nl>
This is done by adding a
```xml
<d:eq>
<d:prop>
<oc:owner-id/>
</d:prop>
<d:literal>$userId</d:literal>
</d:eq>
```
clause to the search query.
Searching by `owner-id` can only be done with the current user id
and the comparison can not be inside a `<d:not>` or `<d:or>` statement
Signed-off-by: Robin Appelman <robin@icewind.nl>
normally this shouldn't be a problem, but cache/storage desync might cause this
so this adds some failsafe to ensure we dont corrupt the cache further
the minimum value is set to -1 instead of 0 in order to triger a background scan
on the folder and figure out the size properly
Signed-off-by: Robin Appelman <robin@icewind.nl>
* Order the imports
* No leading slash on imports
* Empty line before namespace
* One line per import
* Empty after imports
* Emmpty line at bottom of file
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
If you try to do something on a DAV mount (external or federated share)
that is not allowed. We should not mark the storage as not available but
just fail somewhat gracefully.
Now by catching this and just properly returning the operation will just
fail (and notify the user) which is already a lot better then marking
the storage as unavailable and doing boom.
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
Fixes#15048
Catches the case where a full mimetype is sumbitted in the where like
clause. Before we didn't catch this and it was just forwarded as is
causing invalid queries.
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
the target storage doesn't need additional handling for wrappers as the wrappers implementation of moveFromStorage already deals with that
Any storage based on local storage isn't affected by this as local storage already has it's own way of handling with this
Signed-off-by: Robin Appelman <robin@icewind.nl>
korelstar
Stephen Cuppett
Robin Appelman
Roeland Jago Douma
Morris Jobke
Roeland Jago Douma
Georg Ehrke
lui87kw
bladewing
J0WI
Christoph Wurst
Daniel Kesselberg
Florent
Adrian Brzezinski
Scott Dutton
Christoph Wurst
Julien Lutran
Joas Schilling
Julius Härtl
Lukas Stabe
tobiasKaminsky
Greta Doci
Arthur Schiwon