Commit Graph

747 Commits

Author SHA1 Message Date
lui87kw 3fe3d1fca3 Extra white space to follow other ifs
Signed-off-by: lui87kw <lukas.ifflaender@uni-wuerzburg.de>
2020-07-15 10:18:47 +02:00
bladewing 7dafd5831f Avoid substr() error when strpos returns false
"Exception: substr() expects parameter 3 to be int, bool given" can occur on Line 378 $mimePart = substr($icon, 0, strpos($icon, '-'));
This happens, when '-' is not found and strpos returns false instead of an int.
When this occurs, e.g., Activity hangs.

Signed-off-by: lui87kw <lukas.ifflaender@uni-wuerzburg.de>
2020-07-15 10:18:47 +02:00
Robin Appelman e357d46863
filter files containing a hash in the path for ftp storages
the php ftp streamwrapper doesn't handle hashes correctly and will break when it tries to enter a path containing a hash.

By filtering out paths containing a hash we can at least stop the external storage from breaking completely

Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-07-10 14:18:40 +02:00
Robin Appelman 70906a359f
ensure home storage is initialized on first setup
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-07-09 11:14:45 +02:00
Morris Jobke e900f424ce
Merge pull request #21636 from nextcloud/lock-exception-readable-path
add proper paths to locking exceptions
2020-07-05 22:37:12 +02:00
J0WI 356bc50420 Only clear the realpath cache
Signed-off-by: J0WI <J0WI@users.noreply.github.com>
2020-07-03 22:47:13 +02:00
J0WI ea08a01999 clearstatcache() on rmdir
Signed-off-by: J0WI <J0WI@users.noreply.github.com>
2020-07-03 22:42:27 +02:00
Christoph Wurst d3a4bc91f7
Fix static method call for s3 bucket compat check
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-07-03 09:02:58 +02:00
Roeland Jago Douma d53e005458
Merge pull request #21360 from nextcloud/root-storage-prevent-scanning
prevent the root storage from accidentally scanning user folders
2020-07-02 21:07:16 +02:00
Roeland Jago Douma 873be16c8d
Merge pull request #21588 from nextcloud/bug/21214/method-signature
Update presign method to match with interface again.
2020-06-30 20:36:30 +02:00
Robin Appelman da2d425044
add proper paths to locking exceptions
while some code paths do wrap the "raw" locking exception into one with a proper path, not all of them do
by adding the proper path to the original exception we ensure that we always have the usefull information in out logs

Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-06-30 18:10:42 +02:00
Daniel Kesselberg e984357cfc
Set retry = false for forbidden exception.
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2020-06-25 23:34:41 +02:00
Daniel Kesselberg 8aa99aa8ff
Update presign method to match with interface again.
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2020-06-25 22:56:28 +02:00
Robin Appelman 35d75dbb38
relax permissions mask check for detecting part file rename
with files drop uploads the original file name isn't always used for the '.ocTransferId' source path

Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-06-24 13:06:14 +02:00
Christoph Wurst 9d392891be
Disable Client-Side Monitoring on AWS storage
The S3 client enables this by default and then tries to read
`.aws/config`. This causes `open_basedir` restriction related error for
some setups. So this patch disables the CSM because it's most likely
unused anyway.

Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-06-15 09:13:08 +02:00
Robin Appelman e9b5514186
prevent the root storage from accidentally scanning user folders
while this scan *should* never be triggered, it's good to have some failsafe to ensure
that the users home contents don't end up getting scanned in the root storage

Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-06-11 15:41:33 +02:00
Roeland Jago Douma 8bee8e1317
Merge pull request #21317 from FlorentCoppint/s3_upload_part_size
Upload part size as S3 parameter instead of constant value
2020-06-09 19:28:58 +02:00
Florent 3594ba6971 Upload part size as S3 parameter instead of constant value
Some S3 providers need a custom upload part size (500 MB static value in Nextcloud).
Here is a commit to change this value via S3 configuration, instead of using S3_UPLOAD_PART_SIZE constant.
A new parameter is added for an S3 connection : uploadPartSize

Signed-off-by: Florent <florent@coppint.com>
2020-06-09 09:18:42 +02:00
Robin Appelman e75797ad67
increase log level for storage self-test
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-06-08 18:12:49 +02:00
Roeland Jago Douma 6aa6ab3e02
Add lazy events for the Node API
Right now if you want to get events via the Node API you have to have a
real instance of the Root. Which in turns sets up the whole FS.

We should make sure this is done lazy. Else enabling the preview
generator for example makes you setup the whole FS on each and every
authenticated call.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-05-28 12:35:45 +02:00
Roeland Jago Douma 6b26744787
Clear the statscache before fetching the metadata
Else if a lot of writes happen. It might happen that an old stat result
is used. Resulting in a wrong file size for the file. For example the
text app when a lot of people edit at the same time.

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-05-22 09:21:53 +02:00
Robin Appelman f7153a3d0c
generalize the "lazy folder" concept
makes it easy to use lazy intialized folder in other places

Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-05-11 15:36:13 +02:00
Roeland Jago Douma d5850eb28f
Merge pull request #18955 from adrb/swift_upload_large_objects
Large Object support for OpenStack Swift
2020-05-06 13:03:24 +02:00
Robin Appelman 4e6d3d1880
Fix getDirectoryContent implementation for Jail wrapper
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-30 15:27:09 +02:00
Roeland Jago Douma f6b435d0d9
Run the cs fixer for green ci
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-04-30 10:28:44 +02:00
Roeland Jago Douma a1c1b354fc
Merge pull request #18883 from adrb/optimize_swift_14116
Optimize Openstack Swift files download
2020-04-30 09:06:19 +02:00
Christoph Wurst cb057829f7
Update license headers for 19
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-29 11:57:22 +02:00
Adrian Brzezinski e107519295 Large Object support for OpenStack Swift.
Until now, you wouldn't be able to create
objects larger that 5GB.

It's somewhat related with pull #18883

Signed-off-by: Adrian Brzezinski <adrian.brzezinski@eo.pl>
2020-04-27 14:23:59 +02:00
Adrian Brzezinski 354c0996d8 It's a fix for issue #14116,
Improves efficiency when downloading files from Swift storage.
Before, files were downloaded and then pushed back to user.
That behaevior causes all kinds of performance problems.

Now, files are streamed directly to user.

Signed-off-by: Adrian Brzezinski <adrian.brzezinski@eo.pl>
2020-04-27 14:16:50 +02:00
Robin Appelman 436908ee3f
filter out blacklisted files in getDirectoryContent
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-23 13:34:55 +02:00
Robin Appelman 9735b5d61f
Add method to storage backends to get directory content with metadata
Currently you need to use `opendir` and then call `getMetadata` for
every file, which adds overhead because most storage backends already
get the metadata when doing the `opendir`.

While storagebackends can (and do) use caching to relief this problem,
this adds cache invalidation dificulties and only a limited number of
items are generally cached (to prevent memory usage exploding when
scanning large storages)

With this new methods storage backends can use the child metadata they
got from listing the folder to return metadata without having to keep
seperate caches.

Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-20 15:45:04 +02:00
Roeland Jago Douma 79cf13e349
Merge pull request #16946 from exussum12/patch-1
Increase max read
2020-04-16 10:37:10 +02:00
Scott Dutton 65e134c13a
Update View.php 2020-04-15 19:10:06 +01:00
Morris Jobke 7fe8477830
Run cs:fix and fix HashWrapper code
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2020-04-15 14:08:56 +02:00
Morris Jobke d8557d0d8a
Merge pull request #19540 from jlutran/fixKeystoneV3Auth
Fix issues with Keystone auth v3 in files_external app
2020-04-15 14:05:32 +02:00
Christoph Wurst 301327135c
Merge pull request #20361 from nextcloud/write-stream-close-on-exception
Close the streams in `writeStream` even when there is an exception
2020-04-15 10:48:24 +02:00
Julien Lutran 3e849daf11
Fix issues with Keystone auth v3 in files_external app
Signed-off-by: Julien Lutran <julien.lutran@corp.ovh.com>
2020-04-15 10:33:08 +02:00
Robin Appelman c2677682c4
Return hashes of uploaded content for dav uploads
hashes are set in "X-Hash-MD5", "X-Hash-SHA1" and "X-Hash-SHA256" headers.

these headers are set for file uploads and the MOVE request at the end of a multipart upload.

Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-13 12:57:27 +02:00
Scott Dutton 75b99e85ff
Increase max read
8kb is very low, especially given this will be local files

Signed-off-by: Scott Dutton <scott@exussum.co.uk>
2020-04-12 13:01:29 +01:00
Christoph Wurst 734c62bee0
Format code according to PSR2
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:56:50 +02:00
Christoph Wurst 28f8eb5dba
Add visibility to all constants
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:54:27 +02:00
Christoph Wurst 1584c9ae9c
Add visibility to all methods and position of static keyword
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:51:06 +02:00
Christoph Wurst a7c8d26d31
Add visibility to all properties and move static keyword
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 16:48:31 +02:00
Christoph Wurst caff1023ea
Format control structures, classes, methods and function
To continue this formatting madness, here's a tiny patch that adds
unified formatting for control structures like if and loops as well as
classes, their methods and anonymous functions. This basically forces
the constructs to start on the same line. This is not exactly what PSR2
wants, but I think we can have a few exceptions with "our" style. The
starting of braces on the same line is pracrically standard for our
code.

This also removes and empty lines from method/function bodies at the
beginning and end.

Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 14:19:56 +02:00
Christoph Wurst 14c996d982
Use elseif instead of else if
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-10 10:35:09 +02:00
Roeland Jago Douma fb64fda4d8
Merge pull request #20385 from nextcloud/skeeable-http-harden
harden seekable http stream a bit against failures
2020-04-10 08:34:52 +02:00
Christoph Wurst 44577e4345
Remove trailing and in between spaces
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-09 16:07:47 +02:00
Christoph Wurst 42625a46be
Remove spaces after method or function call
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-09 16:05:56 +02:00
Robin Appelman 60de74ac40
harden seekable http stream a bit against failures
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-09 14:37:01 +02:00
Christoph Wurst afbd9c4e6e
Unify function spacing to PSR2 recommendation
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-09 13:54:22 +02:00
Christoph Wurst 41b5e5923a
Use exactly one empty line after the namespace declaration
For PSR2

Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-04-09 11:48:10 +02:00
Robin Appelman ccbf3059ba
Close the streams in `writeStream` even when there is an exception
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-08 16:40:56 +02:00
Joas Schilling f8a5812f98
Try to use the display name of file transfers
Signed-off-by: Joas Schilling <coding@schilljs.com>
2020-04-08 15:41:20 +02:00
Roeland Jago Douma f5919d5b83
Merge pull request #20033 from nextcloud/s3-seekable-stream
Enable fseek for files in S3 storage
2020-04-06 21:38:49 +02:00
Julius Härtl 56aa8fd035
Properly hint exceptions thrown by getUserFolder
Signed-off-by: Julius Härtl <jus@bitgrid.net>
2020-04-03 23:06:26 +02:00
Robin Appelman e22a28ecc2
update autoloader
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-03 16:27:20 +02:00
Robin Appelman 3ba46f3b50
add basic tests for s3 seeking and add some error handling if reopen return the wrong range
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 15:23:05 +02:00
Robin Appelman 7b07e7251c
make seekable s3 stream generic
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 15:21:05 +02:00
Lukas Stabe 14401efb0f
Enable fseek for files in S3 storage
Signed-off-by: Lukas Stabe <lukas@stabe.de>
2020-04-01 15:21:01 +02:00
Robin Appelman d9184584e0
restart transaction when cache insert fails due to duplicate
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 14:17:34 +02:00
Robin Appelman 5199f80196
call cache->insert directly instead of going trough cache->put first when scanning
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 14:17:28 +02:00
Robin Appelman 9c7da26394
dont try to get existing cache data during scanning if we already know its a new file
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 14:17:25 +02:00
Robin Appelman 3fb32f6f3d
reuse known parentId when inserting cache entries in scanner
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 14:14:52 +02:00
Robin Appelman 5439469b68
optimize getMetaData for local storage
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-04-01 14:14:48 +02:00
Christoph Wurst 1a9330cd69
Update the license headers for Nextcloud 19
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-03-31 14:52:54 +02:00
Christoph Wurst 463b388589
Merge pull request #20170 from nextcloud/techdebt/remove-unused-imports
Remove unused imports
2020-03-27 17:14:08 +01:00
Christoph Wurst b80ebc9674
Use the short array syntax, everywhere
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-03-26 16:34:56 +01:00
Christoph Wurst 74936c49ea
Remove unused imports
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-03-25 22:08:08 +01:00
Christoph Wurst 23eb0462e5
Use lowercase true, false and null constants
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2020-03-25 21:53:04 +01:00
Roeland Jago Douma b300dc5a50
Merge pull request #19782 from nextcloud/quota-wrapper-include-external
Use global used space in quota wrappen when external storage is included
2020-03-24 21:36:01 +01:00
Roeland Jago Douma 3443a00621
Actually check if the owner is not null
Else this can error out on storages that do not have an owner (e.g.
groupfolders).

Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-03-16 13:12:10 +01:00
Roeland Jago Douma 9a35e789b4
Get correct mimetype on objectstores
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2020-03-11 11:14:12 +01:00
Roeland Jago Douma b6245be302
Merge pull request #19746 from nextcloud/locked-exception-forward-existing
pass the existing locks info when making locked exception with absolu…
2020-03-09 19:31:45 +01:00
Robin Appelman c32750ddd9
Use global used space in quota wrappen when external storage is included
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-03-04 18:02:50 +01:00
Robin Appelman fab22ac14c
pass the existing locks info when making locked exception with absolute paths
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-03-02 19:29:44 +01:00
Robin Appelman f972990469
dont try to update storage mtime if we can't get the mtime
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-03-02 15:57:25 +01:00
Robin Appelman 63608ef461
allow writing content directly when creating new SimpleFile
Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-02-28 12:55:22 +01:00
Robin Appelman 5ca1929e8c
Create SimpleFile only when writing the content
instead of first creating an empty file and then writing the content.

This solves the overhead of creating an empty file with the common pattern:

```php
$file = $simpleFilder->newFile('foo.txt');
$file->putContent('bar.txt');
```

roughly halving the number of storage and database operations that need to be done when creating a `SimpleFile`.

This is not automatically done with `File` because that has a more complex api which I'm more hesitant to touch.
Instead the `Folder::newFile` api has been extended to accept the content for the new file.

In my local testing, the overhead of first creating an empty file took about 20% of the time for preview generation

Signed-off-by: Robin Appelman <robin@icewind.nl>
2020-02-28 12:55:22 +01:00
Joas Schilling dc050fb5e5
Allow to overwrite the path on the cache event
Signed-off-by: Joas Schilling <coding@schilljs.com>
2020-02-13 08:38:33 +01:00
Joas Schilling 08cadf8416
Don't create invalid users
Signed-off-by: Joas Schilling <coding@schilljs.com>
2020-02-12 10:55:27 +01:00
Christoph Wurst 1b46621cd3
Update license headers for 18
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2019-12-20 09:23:25 +01:00
Christoph Wurst b81b824da1
Add typed events for the filesystem/scanner
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2019-12-13 09:41:05 +01:00
Roeland Jago Douma c6e51924c8
Merge pull request #18236 from nextcloud/bugfix/noid/always-detect-mimetype-by-content-in-workflows
Allow to detect mimetype by content
2019-12-12 11:19:55 +01:00
Roeland Jago Douma 4356c91ffd
Fix detection of non extention types
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-12-12 09:58:22 +01:00
Roeland Jago Douma 4fad9e4e6b
Merge pull request #18314 from nextcloud/fixFileIdSearch
Fix ambigious fileID on search
2019-12-10 11:28:40 +01:00
Joas Schilling 107ef72afb
Try more methods
Signed-off-by: Joas Schilling <coding@schilljs.com>
2019-12-10 10:56:47 +01:00
tobiasKaminsky 41cb214112
fix https://github.com/nextcloud/server/issues/18313
Signed-off-by: tobiasKaminsky <tobias@kaminsky.me>
2019-12-10 09:47:30 +01:00
Joas Schilling da44c2a414
Some more fixes in detecting the mimetype from the content
Signed-off-by: Joas Schilling <coding@schilljs.com>
2019-12-10 09:11:42 +01:00
Joas Schilling 1336dedd5d
Cleanup the code
Signed-off-by: Joas Schilling <coding@schilljs.com>
2019-12-10 09:11:40 +01:00
Joas Schilling b92ebb928a
Allow to check for the mimetype by content only
Signed-off-by: Joas Schilling <coding@schilljs.com>
2019-12-10 09:10:32 +01:00
Roeland Jago Douma 5dba8d318d
Fix search limit to home
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-12-09 20:38:13 +01:00
Daniel Kesselberg 29575c4d36
Move custom definition logic into method
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2019-12-09 12:10:19 +01:00
Daniel Kesselberg 64aba49461
Ensure that we don't merge broken json.
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2019-12-09 12:09:43 +01:00
Christoph Wurst 5bf3d1bb38
Update license headers
Signed-off-by: Christoph Wurst <christoph@winzerhof-wurst.at>
2019-12-05 15:38:45 +01:00
Roeland Jago Douma 63cb31542d
Merge pull request #17941 from nextcloud/search-by-owner
Allow filtering the search results to the users home storage
2019-12-05 11:04:33 +01:00
Roeland Jago Douma 81b919ed9f
Merge pull request #18163 from nextcloud/fix_file_size
fix filename size
2019-12-04 14:15:07 +01:00
Roeland Jago Douma 76b78edd40
Merge pull request #17833 from nextcloud/propagator-no-negative-sizes
dont set folder size to negative values during propagation
2019-12-04 11:49:26 +01:00
Greta Doci 1110a93fc4 fix filename size, add comment
Signed-off-by: Greta Doci <gretadoci@gmail.com>
2019-12-04 11:04:46 +01:00
Robin Appelman f9e5b33c2b
remove old 'owner' filter
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-12-03 15:20:54 +01:00
Robin Appelman c62637da8b
Allow filtering the search results to the users home storage
This is done by adding a

```xml
<d:eq>
    <d:prop>
        <oc:owner-id/>
    </d:prop>
    <d:literal>$userId</d:literal>
</d:eq>
```

clause to the search query.

Searching by `owner-id` can only be done with the current user id
and the comparison can not be inside a `<d:not>` or `<d:or>` statement

Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-12-03 13:49:37 +01:00